[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 18:59:47 dist-test-slave-9fn1 krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 18:59:47 dist-test-slave-9fn1 krb5kdc[395](info): set up 2 sockets
Sep 24 18:59:47 dist-test-slave-9fn1 krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 18:59:48.834138 388 mini_kdc.cc:121] Time spent starting KDC: real 2.180s user 0.010s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 18:59:48 dist-test-slave-9fn1 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T18:59:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T18:59:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 18:59:49.002254 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41039
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41039
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 18:59:49.319410 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 18:59:49.319723 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 18:59:49.319902 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 18:59:49.327303 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 18:59:49.327446 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 18:59:49.327555 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 18:59:49.327656 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 18:59:49.327732 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 18:59:49.337462 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 18:59:49.338831 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 18:59:49.340144 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 18:59:49.352425 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 18:59:49.352358 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 18:59:49.352584 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 18:59:49.353359 408 server_base.cc:1034] running on GCE node
I20260924 18:59:49.354174 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 18:59:49.355584 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 18:59:49.356812 408 hybrid_clock.cc:648] HybridClock initialized: now 1790276389356747 us; error 84 us; skew 500 ppm
I20260924 18:59:49.357161 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 18:59:49 dist-test-slave-9fn1 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 18:59:49.360253 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 18:59:49.361729 408 webserver.cc:460] Webserver started at http://127.0.97.62:34371/ using document root <none> and password file <none>
I20260924 18:59:49.362386 408 fs_manager.cc:359] Metadata directory not provided
I20260924 18:59:49.362537 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 18:59:49.362846 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 18:59:49.365731 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "1e0706878c43465599db8555042b7b0f"
format_stamp: "Formatted at 2026-09-24 18:59:49 on dist-test-slave-9fn1"
server_key: "8dabd205830f0bf990a8083e3afa211a"
server_key_iv: "67113eff901c971b458e2214df6bd1c4"
server_key_version: "encryptionkey@0"
I20260924 18:59:49.367056 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "1e0706878c43465599db8555042b7b0f"
format_stamp: "Formatted at 2026-09-24 18:59:49 on dist-test-slave-9fn1"
server_key: "8dabd205830f0bf990a8083e3afa211a"
server_key_iv: "67113eff901c971b458e2214df6bd1c4"
server_key_version: "encryptionkey@0"
I20260924 18:59:49.372090 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 18:59:49.375319 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 18:59:49.376516 408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 18:59:49.376688 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "1e0706878c43465599db8555042b7b0f"
format_stamp: "Formatted at 2026-09-24 18:59:49 on dist-test-slave-9fn1"
server_key: "8dabd205830f0bf990a8083e3afa211a"
server_key_iv: "67113eff901c971b458e2214df6bd1c4"
server_key_version: "encryptionkey@0"
I20260924 18:59:49.376878 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 18:59:49.394960 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 18:59:49.399431 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 18:59:49.399742 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 18:59:49.417613 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41039
I20260924 18:59:49.417644 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41039 every 8 connection(s)
I20260924 18:59:49.418892 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 18:59:49.422801 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 18:59:49.426358 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 408
I20260924 18:59:49.426678 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 18:59:49.428488 388 external_mini_cluster.cc:1428] Setting key a781f82fa92521d3ba82221410d00b30
I20260924 18:59:49.432785 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e0706878c43465599db8555042b7b0f: Bootstrap starting.
I20260924 18:59:49.435850 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e0706878c43465599db8555042b7b0f: Neither blocks nor log segments found. Creating new log.
I20260924 18:59:49.437186 475 log.cc:824] T 00000000000000000000000000000000 P 1e0706878c43465599db8555042b7b0f: Log is configured to *not* fsync() on all Append() calls
I20260924 18:59:49.439920 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e0706878c43465599db8555042b7b0f: No bootstrap required, opened a new log
I20260924 18:59:49.449383 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e0706878c43465599db8555042b7b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e0706878c43465599db8555042b7b0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41039 } }
I20260924 18:59:49.449853 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e0706878c43465599db8555042b7b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 18:59:49.449961 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e0706878c43465599db8555042b7b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e0706878c43465599db8555042b7b0f, State: Initialized, Role: FOLLOWER
I20260924 18:59:49.450583 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e0706878c43465599db8555042b7b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e0706878c43465599db8555042b7b0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41039 } }
I20260924 18:59:49.450843 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e0706878c43465599db8555042b7b0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 18:59:49.450990 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e0706878c43465599db8555042b7b0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 18:59:49.451153 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e0706878c43465599db8555042b7b0f [term 0 FOLLOWER]: Advancing to term 1
I20260924 18:59:49.453688 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e0706878c43465599db8555042b7b0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e0706878c43465599db8555042b7b0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41039 } }
I20260924 18:59:49.454164 475 leader_election.cc:302] T 00000000000000000000000000000000 P 1e0706878c43465599db8555042b7b0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e0706878c43465599db8555042b7b0f; no voters:
I20260924 18:59:49.454764 475 leader_election.cc:288] T 00000000000000000000000000000000 P 1e0706878c43465599db8555042b7b0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 18:59:49.454955 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e0706878c43465599db8555042b7b0f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 18:59:49.456082 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e0706878c43465599db8555042b7b0f [term 1 LEADER]: Becoming Leader. State: Replica: 1e0706878c43465599db8555042b7b0f, State: Running, Role: LEADER
I20260924 18:59:49.456637 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e0706878c43465599db8555042b7b0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e0706878c43465599db8555042b7b0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41039 } }
I20260924 18:59:49.456914 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e0706878c43465599db8555042b7b0f [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 18:59:49 dist-test-slave-9fn1 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 18:59:49.459858 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e0706878c43465599db8555042b7b0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e0706878c43465599db8555042b7b0f. Latest consensus state: current_term: 1 leader_uuid: "1e0706878c43465599db8555042b7b0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e0706878c43465599db8555042b7b0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41039 } } }
I20260924 18:59:49.460239 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e0706878c43465599db8555042b7b0f [sys.catalog]: This master's current role is: LEADER
I20260924 18:59:49.460758 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e0706878c43465599db8555042b7b0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e0706878c43465599db8555042b7b0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e0706878c43465599db8555042b7b0f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41039 } } }
I20260924 18:59:49.461055 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e0706878c43465599db8555042b7b0f [sys.catalog]: This master's current role is: LEADER
I20260924 18:59:49.463883 486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 18:59:49.470161 486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 18:59:49.474435 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:49.432287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47404 (local address 127.0.97.62:41039)
0924 18:59:49.433182 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:49.433197 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:49.434536 (+ 1339us) server_negotiation.cc:407] Connection header received
0924 18:59:49.444605 (+ 10069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:49.444628 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:49.445127 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:49.445606 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:49.448589 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:49.450524 (+ 1935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:49.452392 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:49.453338 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:49.453543 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:49.461726 (+ 8183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:49.461740 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:49.461763 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:49.461818 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:49.466862 (+ 5044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:49.468218 (+ 1356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:49.468231 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:49.468238 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:49.468450 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:49.469008 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:49.469013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:49.469018 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:49.469696 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:49.470013 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:49.472622 (+ 2609us) server_negotiation.cc:299] Negotiation successful
0924 18:59:49.473048 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":390,"threads_started":1}
I20260924 18:59:49.487299 486 catalog_manager.cc:1347] Generated new cluster ID: 74fe93a5fa0f4ba4b72f5ddbe241a8b1
I20260924 18:59:49.487437 486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 18:59:49.507797 486 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 18:59:49.509173 486 catalog_manager.cc:1504] Loading token signing keys...
I20260924 18:59:49.522823 486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e0706878c43465599db8555042b7b0f: Generated new TSK 0
I20260924 18:59:49.523633 486 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 18:59:49.565270 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41039
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 18:59:49.895691 502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 18:59:49.895972 502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 18:59:49.896070 502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 18:59:49.903373 502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 18:59:49.903478 502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 18:59:49.903589 502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 18:59:49.913223 502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41039
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 18:59:49.914461 502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 18:59:49.915649 502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 18:59:49.928781 507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 18:59:49.928975 510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 18:59:49.929025 508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 18:59:49.930501 502 server_base.cc:1034] running on GCE node
I20260924 18:59:49.931208 502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 18:59:49.932264 502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 18:59:49.933539 502 hybrid_clock.cc:648] HybridClock initialized: now 1790276389933443 us; error 52 us; skew 500 ppm
I20260924 18:59:49.933874 502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 18:59:49 dist-test-slave-9fn1 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 18:59:49.937053 502 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 18:59:49.938627 502 webserver.cc:460] Webserver started at http://127.0.97.1:45993/ using document root <none> and password file <none>
I20260924 18:59:49.939353 502 fs_manager.cc:359] Metadata directory not provided
I20260924 18:59:49.939502 502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 18:59:49.939818 502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 18:59:49.942745 502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-0/data/instance:
uuid: "1061d18d14a246a2a1f2c30b6f0b1084"
format_stamp: "Formatted at 2026-09-24 18:59:49 on dist-test-slave-9fn1"
server_key: "2b4e6f053a723437240e38ad08d5792d"
server_key_iv: "7d6838e601af83bb244447331ce4dfcc"
server_key_version: "encryptionkey@0"
I20260924 18:59:49.943544 502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1061d18d14a246a2a1f2c30b6f0b1084"
format_stamp: "Formatted at 2026-09-24 18:59:49 on dist-test-slave-9fn1"
server_key: "2b4e6f053a723437240e38ad08d5792d"
server_key_iv: "7d6838e601af83bb244447331ce4dfcc"
server_key_version: "encryptionkey@0"
I20260924 18:59:49.948539 502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260924 18:59:49.952242 517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 18:59:49.953449 502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 18:59:49.953658 502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "1061d18d14a246a2a1f2c30b6f0b1084"
format_stamp: "Formatted at 2026-09-24 18:59:49 on dist-test-slave-9fn1"
server_key: "2b4e6f053a723437240e38ad08d5792d"
server_key_iv: "7d6838e601af83bb244447331ce4dfcc"
server_key_version: "encryptionkey@0"
I20260924 18:59:49.953845 502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 18:59:49.975740 502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 18:59:49.980413 502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 18:59:49.980727 502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 18:59:49.981793 502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 18:59:49.983215 502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 18:59:49.983322 502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 18:59:49.983472 502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 18:59:49.983544 502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 18:59:50.017923 502 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32999
I20260924 18:59:50.017957 629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32999 every 8 connection(s)
I20260924 18:59:50.019235 502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 18:59:50.021147 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 502
I20260924 18:59:50.021359 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-0/wal/instance
I20260924 18:59:50.022200 388 external_mini_cluster.cc:1428] Setting key 0164452f10581e1d0e24128722ff5307
Sep 24 18:59:50 dist-test-slave-9fn1 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260924 18:59:50.042908 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:50.021926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42213 (local address 127.0.97.62:41039)
0924 18:59:50.022608 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:50.022615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:50.023562 (+ 947us) server_negotiation.cc:407] Connection header received
0924 18:59:50.026066 (+ 2504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:50.026070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:50.026130 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:50.026220 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:50.028690 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:50.029600 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:50.031147 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:50.031936 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:50.032075 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:50.035603 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:50.035610 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:50.035616 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:50.035659 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:50.039657 (+ 3998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:50.040433 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:50.040437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:50.040440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:50.040554 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:50.041041 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:50.041044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:50.041046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:50.041309 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:50.041428 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:50.042203 (+ 775us) server_negotiation.cc:299] Negotiation successful
0924 18:59:50.042462 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":439,"threads_started":1}
I20260924 18:59:50.044085 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:50.022564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41039 (local address 127.0.97.1:42213)
0924 18:59:50.023403 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:50.023453 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:50.025772 (+ 2319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:50.026498 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:50.026512 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:50.027745 (+ 1233us) client_negotiation.cc:189] Negotiated authn=SASL
0924 18:59:50.028524 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:50.028550 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:50.029784 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:50.029789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:50.030996 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:50.031008 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:50.032147 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:50.032151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:50.032348 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:50.033060 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 18:59:50.033086 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 18:59:50.035365 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 18:59:50.039803 (+ 4438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 18:59:50.039810 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 18:59:50.039825 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 18:59:50.040311 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 18:59:50.040712 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 18:59:50.040716 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 18:59:50.040718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 18:59:50.040905 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 18:59:50.041421 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 18:59:50.041427 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 18:59:50.041872 (+ 445us) client_negotiation.cc:769] Sending connection context
0924 18:59:50.042120 (+ 248us) client_negotiation.cc:240] Negotiation successful
0924 18:59:50.042621 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":326,"threads_started":1}
I20260924 18:59:50.045677 630 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41039
I20260924 18:59:50.045979 630 heartbeater.cc:464] Registering TS with master...
I20260924 18:59:50.046662 630 heartbeater.cc:511] Master 127.0.97.62:41039 requested a full tablet report, sending...
I20260924 18:59:50.048813 440 ts_manager.cc:194] Registered new tserver with Master: 1061d18d14a246a2a1f2c30b6f0b1084 (127.0.97.1:32999)
I20260924 18:59:50.050303 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42213
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 18:59:50.065893 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41039
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 18:59:50.388672 636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 18:59:50.388950 636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 18:59:50.389109 636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 18:59:50.396198 636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 18:59:50.396313 636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 18:59:50.396497 636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 18:59:50.406898 636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41039
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 18:59:50.408157 636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 18:59:50.409446 636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 18:59:50.428011 644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 18:59:50.428228 642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 18:59:50.428061 641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 18:59:50.430130 636 server_base.cc:1034] running on GCE node
I20260924 18:59:50.430764 636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 18:59:50.431790 636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 18:59:50.433043 636 hybrid_clock.cc:648] HybridClock initialized: now 1790276390432998 us; error 66 us; skew 500 ppm
I20260924 18:59:50.433384 636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 18:59:50 dist-test-slave-9fn1 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 18:59:50.437657 636 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 18:59:50.442230 636 webserver.cc:460] Webserver started at http://127.0.97.2:41329/ using document root <none> and password file <none>
I20260924 18:59:50.443307 636 fs_manager.cc:359] Metadata directory not provided
I20260924 18:59:50.443449 636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 18:59:50.443765 636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 18:59:50.446782 636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-1/data/instance:
uuid: "c081dc2343244743a6f7a18378d8ff79"
format_stamp: "Formatted at 2026-09-24 18:59:50 on dist-test-slave-9fn1"
server_key: "d0d029bc600b1361670cfece9977f0df"
server_key_iv: "bd3dc5ba2abc5570c847f521b1dd3c74"
server_key_version: "encryptionkey@0"
I20260924 18:59:50.447537 636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c081dc2343244743a6f7a18378d8ff79"
format_stamp: "Formatted at 2026-09-24 18:59:50 on dist-test-slave-9fn1"
server_key: "d0d029bc600b1361670cfece9977f0df"
server_key_iv: "bd3dc5ba2abc5570c847f521b1dd3c74"
server_key_version: "encryptionkey@0"
I20260924 18:59:50.452560 636 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260924 18:59:50.455978 651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 18:59:50.457381 636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 18:59:50.457605 636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "c081dc2343244743a6f7a18378d8ff79"
format_stamp: "Formatted at 2026-09-24 18:59:50 on dist-test-slave-9fn1"
server_key: "d0d029bc600b1361670cfece9977f0df"
server_key_iv: "bd3dc5ba2abc5570c847f521b1dd3c74"
server_key_version: "encryptionkey@0"
I20260924 18:59:50.457844 636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 18:59:50.471462 636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 18:59:50.475219 636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 18:59:50.475497 636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 18:59:50.476760 636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 18:59:50.478081 636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 18:59:50.478184 636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 18:59:50.478318 636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 18:59:50.478391 636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 18:59:50.513101 636 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45375
I20260924 18:59:50.513157 763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45375 every 8 connection(s)
I20260924 18:59:50.514345 636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 18:59:50.521898 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 636
I20260924 18:59:50.522096 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-1/wal/instance
I20260924 18:59:50.522641 388 external_mini_cluster.cc:1428] Setting key fafa03964a21394b4d26d4e4b35ddaf5
Sep 24 18:59:50 dist-test-slave-9fn1 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 18:59:50.535750 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:50.516883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59019 (local address 127.0.97.62:41039)
0924 18:59:50.517052 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:50.517057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:50.518128 (+ 1071us) server_negotiation.cc:407] Connection header received
0924 18:59:50.519823 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:50.519827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:50.519890 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:50.519976 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:50.521940 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:50.523044 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:50.524829 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:50.525487 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:50.525640 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:50.528867 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:50.528874 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:50.528880 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:50.528921 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:50.532496 (+ 3575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:50.533178 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:50.533186 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:50.533193 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:50.533359 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:50.533859 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:50.533868 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:50.533874 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:50.534251 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:50.534429 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:50.535065 (+ 636us) server_negotiation.cc:299] Negotiation successful
0924 18:59:50.535332 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 18:59:50.536425 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:50.517246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41039 (local address 127.0.97.2:59019)
0924 18:59:50.517972 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:50.518009 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:50.519581 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:50.520182 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:50.520189 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:50.520679 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0924 18:59:50.521770 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:50.521789 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:50.523182 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:50.523187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:50.524650 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:50.524661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:50.525688 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:50.525693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:50.525876 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:50.526546 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 18:59:50.526566 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 18:59:50.528652 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 18:59:50.532640 (+ 3988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 18:59:50.532646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 18:59:50.532676 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0924 18:59:50.533053 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 18:59:50.533466 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 18:59:50.533469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 18:59:50.533471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 18:59:50.533744 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 18:59:50.534386 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 18:59:50.534392 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 18:59:50.534703 (+ 311us) client_negotiation.cc:769] Sending connection context
0924 18:59:50.534916 (+ 213us) client_negotiation.cc:240] Negotiation successful
0924 18:59:50.535355 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":313,"threads_started":1}
I20260924 18:59:50.537936 764 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41039
I20260924 18:59:50.538244 764 heartbeater.cc:464] Registering TS with master...
I20260924 18:59:50.538931 764 heartbeater.cc:511] Master 127.0.97.62:41039 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 18:59:50.540422 440 ts_manager.cc:194] Registered new tserver with Master: c081dc2343244743a6f7a18378d8ff79 (127.0.97.2:45375)
I20260924 18:59:50.541275 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 18:59:50.562443 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41039
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 18:59:50.888496 769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 18:59:50.888768 769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 18:59:50.888918 769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 18:59:50.896670 769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 18:59:50.896868 769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 18:59:50.897084 769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 18:59:50.907207 769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41039
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 18:59:50.908447 769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 18:59:50.909781 769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 18:59:50.925997 775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 18:59:50.926062 774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 18:59:50.926023 777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 18:59:50.926681 769 server_base.cc:1034] running on GCE node
I20260924 18:59:50.927275 769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 18:59:50.928145 769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 18:59:50.929405 769 hybrid_clock.cc:648] HybridClock initialized: now 1790276390929374 us; error 70 us; skew 500 ppm
I20260924 18:59:50.929759 769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 18:59:50 dist-test-slave-9fn1 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 18:59:50.932721 769 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 18:59:50.934613 769 webserver.cc:460] Webserver started at http://127.0.97.3:44087/ using document root <none> and password file <none>
I20260924 18:59:50.935324 769 fs_manager.cc:359] Metadata directory not provided
I20260924 18:59:50.935468 769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 18:59:50.935781 769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 18:59:50.938764 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-2/data/instance:
uuid: "59d9b0e78d104dea8f21ed20ba187c7a"
format_stamp: "Formatted at 2026-09-24 18:59:50 on dist-test-slave-9fn1"
server_key: "00b106d75766017915a62bd0827c4f64"
server_key_iv: "7a54fe9c4e737320cd883b9ee5862882"
server_key_version: "encryptionkey@0"
I20260924 18:59:50.939560 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-2/wal/instance:
uuid: "59d9b0e78d104dea8f21ed20ba187c7a"
format_stamp: "Formatted at 2026-09-24 18:59:50 on dist-test-slave-9fn1"
server_key: "00b106d75766017915a62bd0827c4f64"
server_key_iv: "7a54fe9c4e737320cd883b9ee5862882"
server_key_version: "encryptionkey@0"
I20260924 18:59:50.944603 769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 18:59:50.948041 784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 18:59:50.949324 769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 18:59:50.949527 769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "59d9b0e78d104dea8f21ed20ba187c7a"
format_stamp: "Formatted at 2026-09-24 18:59:50 on dist-test-slave-9fn1"
server_key: "00b106d75766017915a62bd0827c4f64"
server_key_iv: "7a54fe9c4e737320cd883b9ee5862882"
server_key_version: "encryptionkey@0"
I20260924 18:59:50.949756 769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 18:59:50.962098 769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 18:59:50.966238 769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 18:59:50.966523 769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 18:59:50.967340 769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 18:59:50.968508 769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 18:59:50.968576 769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 18:59:50.968657 769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 18:59:50.968703 769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 18:59:51.001974 769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37179
I20260924 18:59:51.002015 896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37179 every 8 connection(s)
I20260924 18:59:51.003185 769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 18:59:51.007392 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 769
I20260924 18:59:51.007573 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276386554690-388-0/minicluster-data/ts-2/wal/instance
I20260924 18:59:51.008004 388 external_mini_cluster.cc:1428] Setting key 2a9b2cfd7d4c2b533f8c01faa856654e
Sep 24 18:59:51 dist-test-slave-9fn1 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 18:59:51.024250 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:51.005697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43369 (local address 127.0.97.62:41039)
0924 18:59:51.005904 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:51.005909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:51.007096 (+ 1187us) server_negotiation.cc:407] Connection header received
0924 18:59:51.009661 (+ 2565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:51.009665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:51.009755 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:51.009893 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:51.011629 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:51.012342 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:51.013912 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:51.014274 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:51.014399 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:51.017854 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:51.017860 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:51.017865 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:51.017905 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:51.021116 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:51.021880 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:51.021889 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:51.021892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:51.021999 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:51.022467 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:51.022471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:51.022474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:51.022868 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:51.023018 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:51.023864 (+ 846us) server_negotiation.cc:299] Negotiation successful
0924 18:59:51.024013 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 18:59:51.025075 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:51.006268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41039 (local address 127.0.97.3:43369)
0924 18:59:51.006918 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:51.006963 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:51.009344 (+ 2381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:51.010129 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:51.010137 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:51.010817 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0924 18:59:51.011477 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:51.011494 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:51.012500 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:51.012510 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:51.013783 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:51.013794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:51.014474 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:51.014483 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:51.014705 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:51.015362 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 18:59:51.015383 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 18:59:51.017644 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 18:59:51.021266 (+ 3622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 18:59:51.021272 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 18:59:51.021296 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 18:59:51.021741 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 18:59:51.022131 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 18:59:51.022136 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 18:59:51.022139 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 18:59:51.022367 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 18:59:51.022990 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 18:59:51.022997 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 18:59:51.023391 (+ 394us) client_negotiation.cc:769] Sending connection context
0924 18:59:51.023646 (+ 255us) client_negotiation.cc:240] Negotiation successful
0924 18:59:51.024068 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":256,"threads_started":1}
I20260924 18:59:51.026319 897 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41039
I20260924 18:59:51.026602 897 heartbeater.cc:464] Registering TS with master...
I20260924 18:59:51.027279 897 heartbeater.cc:511] Master 127.0.97.62:41039 requested a full tablet report, sending...
I20260924 18:59:51.028524 440 ts_manager.cc:194] Registered new tserver with Master: 59d9b0e78d104dea8f21ed20ba187c7a (127.0.97.3:37179)
I20260924 18:59:51.029277 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43369
I20260924 18:59:51.035542 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 18:59:51.054124 630 heartbeater.cc:503] Master 127.0.97.62:41039 was elected leader, sending a full tablet report...
Sep 24 18:59:51 dist-test-slave-9fn1 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276391, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 18:59:51 dist-test-slave-9fn1 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276391, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 18:59:51.077982 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:51.063329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57824 (local address 127.0.97.1:32999)
0924 18:59:51.064004 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:51.064013 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:51.064042 (+ 29us) server_negotiation.cc:407] Connection header received
0924 18:59:51.064187 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:51.064194 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:51.064391 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:51.064556 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:51.065935 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:51.066916 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:51.068154 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:51.068690 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:51.068815 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:51.072647 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:51.072657 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:51.072670 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:51.072697 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:51.075689 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:51.076267 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:51.076273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:51.076275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:51.076383 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:51.076720 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:51.076723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:51.076725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:51.077030 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:51.077249 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:51.077478 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 18:59:51.077714 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":336,"threads_started":1}
W20260924 18:59:51.079185 545 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57824
Sep 24 18:59:51 dist-test-slave-9fn1 krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 18:59:51 dist-test-slave-9fn1 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 18:59:51.119431 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:51.107058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57834 (local address 127.0.97.1:32999)
0924 18:59:51.107303 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:51.107308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:51.107397 (+ 89us) server_negotiation.cc:407] Connection header received
0924 18:59:51.107539 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:51.107543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:51.107632 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:51.107716 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:51.108934 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:51.109674 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:51.110826 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:51.111215 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:51.111307 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:51.114923 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:51.114928 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:51.114932 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:51.114962 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:51.117295 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:51.117918 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:51.117930 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:51.117932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:51.118041 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:51.118333 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:51.118336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:51.118338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:51.118556 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:51.118720 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:51.118964 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 18:59:51.119111 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 18:59:51.121263 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 502
I20260924 18:59:51.132791 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:51.122202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57836 (local address 127.0.97.1:32999)
0924 18:59:51.122342 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:51.122346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:51.122748 (+ 402us) server_negotiation.cc:407] Connection header received
0924 18:59:51.122861 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:51.122865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:51.122913 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:51.122985 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:51.124053 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:51.124713 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:51.125986 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:51.126373 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:51.126463 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:51.128450 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:51.128455 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:51.128458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:51.128483 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:51.130764 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:51.131289 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:51.131292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:51.131294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:51.131403 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:51.131684 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:51.131687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:51.131689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:51.131902 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:51.132062 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:51.132275 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 18:59:51.132415 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 18:59:51.133430 625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 18:59:51.214551 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 502
I20260924 18:59:51.230274 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 636
Sep 24 18:59:51 dist-test-slave-9fn1 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 18:59:51.245456 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:51.231153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57164 (local address 127.0.97.2:45375)
0924 18:59:51.231647 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:51.231654 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:51.231679 (+ 25us) server_negotiation.cc:407] Connection header received
0924 18:59:51.231737 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:51.231744 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:51.231951 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:51.232122 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:51.233286 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:51.234390 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:51.235614 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:51.236148 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:51.236295 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:51.240223 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:51.240234 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:51.240248 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:51.240278 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:51.243195 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:51.243732 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:51.243739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:51.243741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:51.243847 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:51.244182 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:51.244185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:51.244187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:51.244477 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:51.244713 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:51.244901 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 18:59:51.245088 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":252,"threads_started":1}
I20260924 18:59:51.246759 759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 18:59:51.328101 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 636
I20260924 18:59:51.350739 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 769
Sep 24 18:59:51 dist-test-slave-9fn1 krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 18:59:51.366266 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:51.351686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44654 (local address 127.0.97.3:37179)
0924 18:59:51.352285 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:51.352292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:51.352320 (+ 28us) server_negotiation.cc:407] Connection header received
0924 18:59:51.352378 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:51.352384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:51.352599 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:51.352725 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:51.354038 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:51.355049 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:51.356229 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:51.356796 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:51.356888 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:51.360899 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:51.360909 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:51.360922 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:51.360955 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:51.363551 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:51.364177 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:51.364184 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:51.364186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:51.364304 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:51.364663 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:51.364669 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:51.364672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:51.364988 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:51.365289 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:51.365714 (+ 425us) server_negotiation.cc:299] Negotiation successful
0924 18:59:51.365965 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":278,"threads_started":1}
I20260924 18:59:51.367542 892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 18:59:51.448879 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 769
I20260924 18:59:51.464727 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 408
I20260924 18:59:51.465986 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 18:59:51.565376 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 408
2026-09-24T18:59:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4963 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 18:59:51 dist-test-slave-9fn1 krb5kdc[918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 18:59:51 dist-test-slave-9fn1 krb5kdc[918](info): set up 2 sockets
Sep 24 18:59:51 dist-test-slave-9fn1 krb5kdc[918](info): commencing operation
krb5kdc: starting...
W20260924 18:59:52.633256 388 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 18:59:52 dist-test-slave-9fn1 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T18:59:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T18:59:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 18:59:52.798820 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33979
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 18:59:53.102865 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 18:59:53.103119 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 18:59:53.103272 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 18:59:53.110321 931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 18:59:53.110423 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 18:59:53.110512 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 18:59:53.110610 931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 18:59:53.110689 931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 18:59:53.119880 931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 18:59:53.121049 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 18:59:53.122378 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 18:59:53.133771 939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 18:59:53.133785 936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 18:59:53.133787 937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 18:59:53.134648 931 server_base.cc:1034] running on GCE node
I20260924 18:59:53.135303 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 18:59:53.136590 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 18:59:53.137794 931 hybrid_clock.cc:648] HybridClock initialized: now 1790276393137745 us; error 44 us; skew 500 ppm
I20260924 18:59:53.138127 931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 18:59:53 dist-test-slave-9fn1 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 18:59:53.145547 931 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 18:59:53.147004 931 webserver.cc:460] Webserver started at http://127.0.97.62:46239/ using document root <none> and password file <none>
I20260924 18:59:53.147663 931 fs_manager.cc:359] Metadata directory not provided
I20260924 18:59:53.147787 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 18:59:53.148115 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 18:59:53.151010 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "03fbba6bc2e04528a151924c58fe4a53"
format_stamp: "Formatted at 2026-09-24 18:59:53 on dist-test-slave-9fn1"
server_key: "cd9898b075679c2158c5e55d5b4f53a9"
server_key_iv: "c9b6365669841639d85d38a161f47203"
server_key_version: "encryptionkey@0"
I20260924 18:59:53.151729 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "03fbba6bc2e04528a151924c58fe4a53"
format_stamp: "Formatted at 2026-09-24 18:59:53 on dist-test-slave-9fn1"
server_key: "cd9898b075679c2158c5e55d5b4f53a9"
server_key_iv: "c9b6365669841639d85d38a161f47203"
server_key_version: "encryptionkey@0"
I20260924 18:59:53.156525 931 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 18:59:53.159688 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 18:59:53.160910 931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 18:59:53.161080 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "03fbba6bc2e04528a151924c58fe4a53"
format_stamp: "Formatted at 2026-09-24 18:59:53 on dist-test-slave-9fn1"
server_key: "cd9898b075679c2158c5e55d5b4f53a9"
server_key_iv: "c9b6365669841639d85d38a161f47203"
server_key_version: "encryptionkey@0"
I20260924 18:59:53.161269 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 18:59:53.184901 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 18:59:53.188745 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 18:59:53.189038 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 18:59:53.206343 931 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33979
I20260924 18:59:53.206378 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33979 every 8 connection(s)
I20260924 18:59:53.207664 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 18:59:53.211241 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 18:59:53.214003 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 931
I20260924 18:59:53.214222 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 18:59:53.215039 388 external_mini_cluster.cc:1428] Setting key e7b2b29a5f4db60b72efcf7771657983
I20260924 18:59:53.223258 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03fbba6bc2e04528a151924c58fe4a53: Bootstrap starting.
I20260924 18:59:53.226928 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03fbba6bc2e04528a151924c58fe4a53: Neither blocks nor log segments found. Creating new log.
I20260924 18:59:53.228233 998 log.cc:824] T 00000000000000000000000000000000 P 03fbba6bc2e04528a151924c58fe4a53: Log is configured to *not* fsync() on all Append() calls
Sep 24 18:59:53 dist-test-slave-9fn1 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 18:59:53.231034 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03fbba6bc2e04528a151924c58fe4a53: No bootstrap required, opened a new log
I20260924 18:59:53.237161 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:53.217032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38996 (local address 127.0.97.62:33979)
0924 18:59:53.217793 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:53.217804 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:53.217841 (+ 37us) server_negotiation.cc:407] Connection header received
0924 18:59:53.219367 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:53.219387 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:53.219742 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:53.220037 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:53.221775 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:53.223017 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:53.224744 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:53.225374 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:53.225494 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:53.230207 (+ 4713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:53.230215 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:53.230228 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:53.230258 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:53.233253 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:53.233993 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:53.234001 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:53.234014 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:53.234216 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:53.234576 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:53.234581 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:53.234584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:53.235086 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:53.235409 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:53.235855 (+ 446us) server_negotiation.cc:299] Negotiation successful
0924 18:59:53.236223 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":361,"threads_started":1}
I20260924 18:59:53.238847 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03fbba6bc2e04528a151924c58fe4a53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03fbba6bc2e04528a151924c58fe4a53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33979 } }
I20260924 18:59:53.239212 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03fbba6bc2e04528a151924c58fe4a53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 18:59:53.239331 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03fbba6bc2e04528a151924c58fe4a53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03fbba6bc2e04528a151924c58fe4a53, State: Initialized, Role: FOLLOWER
I20260924 18:59:53.239920 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03fbba6bc2e04528a151924c58fe4a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03fbba6bc2e04528a151924c58fe4a53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33979 } }
I20260924 18:59:53.240154 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03fbba6bc2e04528a151924c58fe4a53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 18:59:53.240284 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03fbba6bc2e04528a151924c58fe4a53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 18:59:53.240454 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03fbba6bc2e04528a151924c58fe4a53 [term 0 FOLLOWER]: Advancing to term 1
I20260924 18:59:53.242558 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03fbba6bc2e04528a151924c58fe4a53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03fbba6bc2e04528a151924c58fe4a53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33979 } }
I20260924 18:59:53.243008 998 leader_election.cc:302] T 00000000000000000000000000000000 P 03fbba6bc2e04528a151924c58fe4a53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03fbba6bc2e04528a151924c58fe4a53; no voters:
I20260924 18:59:53.243654 998 leader_election.cc:288] T 00000000000000000000000000000000 P 03fbba6bc2e04528a151924c58fe4a53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 18:59:53.243752 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03fbba6bc2e04528a151924c58fe4a53 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 18:59:53.244421 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03fbba6bc2e04528a151924c58fe4a53 [term 1 LEADER]: Becoming Leader. State: Replica: 03fbba6bc2e04528a151924c58fe4a53, State: Running, Role: LEADER
I20260924 18:59:53.245067 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03fbba6bc2e04528a151924c58fe4a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03fbba6bc2e04528a151924c58fe4a53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33979 } }
I20260924 18:59:53.246368 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03fbba6bc2e04528a151924c58fe4a53 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 18:59:53.247865 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03fbba6bc2e04528a151924c58fe4a53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03fbba6bc2e04528a151924c58fe4a53. Latest consensus state: current_term: 1 leader_uuid: "03fbba6bc2e04528a151924c58fe4a53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03fbba6bc2e04528a151924c58fe4a53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33979 } } }
I20260924 18:59:53.247951 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03fbba6bc2e04528a151924c58fe4a53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03fbba6bc2e04528a151924c58fe4a53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03fbba6bc2e04528a151924c58fe4a53" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33979 } } }
I20260924 18:59:53.248346 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03fbba6bc2e04528a151924c58fe4a53 [sys.catalog]: This master's current role is: LEADER
I20260924 18:59:53.248373 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03fbba6bc2e04528a151924c58fe4a53 [sys.catalog]: This master's current role is: LEADER
I20260924 18:59:53.249747 1008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 18:59:53.254711 1008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 18:59:53.262742 1008 catalog_manager.cc:1347] Generated new cluster ID: cefcce4d81334b628635f5efd3195af2
I20260924 18:59:53.262859 1008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 18:59:53.278569 1008 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 18:59:53.280037 1008 catalog_manager.cc:1504] Loading token signing keys...
I20260924 18:59:53.289235 1008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03fbba6bc2e04528a151924c58fe4a53: Generated new TSK 0
I20260924 18:59:53.290136 1008 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 18:59:53.329453 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33979
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 18:59:53.643052 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 18:59:53.643329 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 18:59:53.643424 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 18:59:53.650767 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 18:59:53.650869 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 18:59:53.651027 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 18:59:53.660374 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33979
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 18:59:53.661518 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 18:59:53.662740 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 18:59:53.675360 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 18:59:53.675310 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 18:59:53.675781 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 18:59:53.676141 1024 server_base.cc:1034] running on GCE node
I20260924 18:59:53.676821 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 18:59:53.677862 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 18:59:53.679088 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790276393679036 us; error 55 us; skew 500 ppm
I20260924 18:59:53.679409 1024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 18:59:53 dist-test-slave-9fn1 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 18:59:53.682469 1024 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 18:59:53.683929 1024 webserver.cc:460] Webserver started at http://127.0.97.1:45411/ using document root <none> and password file <none>
I20260924 18:59:53.684553 1024 fs_manager.cc:359] Metadata directory not provided
I20260924 18:59:53.684648 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 18:59:53.684899 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 18:59:53.687674 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-0/data/instance:
uuid: "63af947c26d24297ba280d4ab33dfd54"
format_stamp: "Formatted at 2026-09-24 18:59:53 on dist-test-slave-9fn1"
server_key: "eaabf42745a796574665a0207d6f98c9"
server_key_iv: "6aa49ed2c65674cc6627fa09c0561929"
server_key_version: "encryptionkey@0"
I20260924 18:59:53.688362 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-0/wal/instance:
uuid: "63af947c26d24297ba280d4ab33dfd54"
format_stamp: "Formatted at 2026-09-24 18:59:53 on dist-test-slave-9fn1"
server_key: "eaabf42745a796574665a0207d6f98c9"
server_key_iv: "6aa49ed2c65674cc6627fa09c0561929"
server_key_version: "encryptionkey@0"
I20260924 18:59:53.693002 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 18:59:53.696290 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 18:59:53.697810 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 18:59:53.697978 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "63af947c26d24297ba280d4ab33dfd54"
format_stamp: "Formatted at 2026-09-24 18:59:53 on dist-test-slave-9fn1"
server_key: "eaabf42745a796574665a0207d6f98c9"
server_key_iv: "6aa49ed2c65674cc6627fa09c0561929"
server_key_version: "encryptionkey@0"
I20260924 18:59:53.698180 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 18:59:53.719473 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 18:59:53.723587 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 18:59:53.723878 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 18:59:53.724843 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 18:59:53.727130 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 18:59:53.727234 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 18:59:53.727411 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 18:59:53.727481 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 18:59:53.760111 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40019
I20260924 18:59:53.760162 1151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40019 every 8 connection(s)
I20260924 18:59:53.761417 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 18:59:53.764902 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 1024
I20260924 18:59:53.765118 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-0/wal/instance
I20260924 18:59:53.765954 388 external_mini_cluster.cc:1428] Setting key c081de0d6f8dbc7d6c4f8a0a5745b2e3
Sep 24 18:59:53 dist-test-slave-9fn1 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 18:59:53.783896 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:53.764009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57515 (local address 127.0.97.62:33979)
0924 18:59:53.764445 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:53.764451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:53.765605 (+ 1154us) server_negotiation.cc:407] Connection header received
0924 18:59:53.767644 (+ 2039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:53.767651 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:53.767728 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:53.767847 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:53.769920 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:53.770603 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:53.772851 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:53.773601 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:53.773732 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:53.776999 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:53.777008 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:53.777016 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:53.777059 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:53.780801 (+ 3742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:53.781565 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:53.781572 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:53.781576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:53.781717 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:53.782159 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:53.782165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:53.782169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:53.782569 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:53.782717 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:53.783435 (+ 718us) server_negotiation.cc:299] Negotiation successful
0924 18:59:53.783650 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":193,"threads_started":1}
I20260924 18:59:53.785065 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:53.764600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33979 (local address 127.0.97.1:57515)
0924 18:59:53.765429 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:53.765478 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:53.767353 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:53.768105 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:53.768116 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:53.768881 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0924 18:59:53.769730 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:53.769759 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:53.770759 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:53.770767 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:53.772718 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:53.772729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:53.773816 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:53.773821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:53.774002 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:53.774707 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 18:59:53.774742 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 18:59:53.776770 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 18:59:53.780952 (+ 4182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 18:59:53.780959 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 18:59:53.780971 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 18:59:53.781424 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 18:59:53.781840 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 18:59:53.781843 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 18:59:53.781846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 18:59:53.782037 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 18:59:53.782715 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 18:59:53.782733 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 18:59:53.783031 (+ 298us) client_negotiation.cc:769] Sending connection context
0924 18:59:53.783332 (+ 301us) client_negotiation.cc:240] Negotiation successful
0924 18:59:53.783738 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":335,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260924 18:59:53.786844 1152 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33979
I20260924 18:59:53.787150 1152 heartbeater.cc:464] Registering TS with master...
I20260924 18:59:53.787823 1152 heartbeater.cc:511] Master 127.0.97.62:33979 requested a full tablet report, sending...
I20260924 18:59:53.790130 963 ts_manager.cc:194] Registered new tserver with Master: 63af947c26d24297ba280d4ab33dfd54 (127.0.97.1:40019)
I20260924 18:59:53.791679 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 18:59:53.808059 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33979
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 18:59:54.113085 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 18:59:54.113358 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 18:59:54.113451 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 18:59:54.120517 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 18:59:54.120640 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 18:59:54.120750 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 18:59:54.130162 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33979
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 18:59:54.131357 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 18:59:54.132598 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 18:59:54.145099 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 18:59:54.145046 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 18:59:54.145155 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 18:59:54.146121 1158 server_base.cc:1034] running on GCE node
I20260924 18:59:54.146615 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 18:59:54.147534 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 18:59:54.148720 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790276394148688 us; error 30 us; skew 500 ppm
I20260924 18:59:54.148999 1158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 18:59:54 dist-test-slave-9fn1 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 18:59:54.152063 1158 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 18:59:54.153383 1158 webserver.cc:460] Webserver started at http://127.0.97.2:41913/ using document root <none> and password file <none>
I20260924 18:59:54.154021 1158 fs_manager.cc:359] Metadata directory not provided
I20260924 18:59:54.154112 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 18:59:54.154345 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 18:59:54.156937 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-1/data/instance:
uuid: "86641b1adeed48219b4d4af1f733d9fc"
format_stamp: "Formatted at 2026-09-24 18:59:54 on dist-test-slave-9fn1"
server_key: "7e19ebbedbcf263da8885f57d8b02ecb"
server_key_iv: "09224ee9d8ffb87eef488ed0e23b016c"
server_key_version: "encryptionkey@0"
I20260924 18:59:54.157692 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-1/wal/instance:
uuid: "86641b1adeed48219b4d4af1f733d9fc"
format_stamp: "Formatted at 2026-09-24 18:59:54 on dist-test-slave-9fn1"
server_key: "7e19ebbedbcf263da8885f57d8b02ecb"
server_key_iv: "09224ee9d8ffb87eef488ed0e23b016c"
server_key_version: "encryptionkey@0"
I20260924 18:59:54.162513 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 18:59:54.165882 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 18:59:54.167101 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 18:59:54.167284 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "86641b1adeed48219b4d4af1f733d9fc"
format_stamp: "Formatted at 2026-09-24 18:59:54 on dist-test-slave-9fn1"
server_key: "7e19ebbedbcf263da8885f57d8b02ecb"
server_key_iv: "09224ee9d8ffb87eef488ed0e23b016c"
server_key_version: "encryptionkey@0"
I20260924 18:59:54.167471 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 18:59:54.200373 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 18:59:54.204421 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 18:59:54.204751 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 18:59:54.205771 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 18:59:54.207157 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 18:59:54.207227 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 18:59:54.207369 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 18:59:54.207422 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 18:59:54.239629 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46087
I20260924 18:59:54.239669 1285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46087 every 8 connection(s)
I20260924 18:59:54.240840 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 18:59:54.243647 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 1158
I20260924 18:59:54.243830 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-1/wal/instance
I20260924 18:59:54.244251 388 external_mini_cluster.cc:1428] Setting key 5433c194f1e50c1782a2757df29a04e1
Sep 24 18:59:54 dist-test-slave-9fn1 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 18:59:54.260341 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:54.243234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45943 (local address 127.0.97.62:33979)
0924 18:59:54.243416 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:54.243423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:54.244658 (+ 1235us) server_negotiation.cc:407] Connection header received
0924 18:59:54.246725 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:54.246730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:54.246828 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:54.246953 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:54.248445 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.249159 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.250775 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.251151 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.251281 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:54.254728 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:54.254734 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:54.254747 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:54.254786 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:54.257435 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:54.258154 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:54.258159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:54.258163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:54.258302 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:54.258708 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:54.258712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:54.258716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:54.258959 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:54.259107 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:54.259801 (+ 694us) server_negotiation.cc:299] Negotiation successful
0924 18:59:54.260017 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 18:59:54.261385 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:54.243643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33979 (local address 127.0.97.2:45943)
0924 18:59:54.244482 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:54.244536 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:54.246474 (+ 1938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:54.247215 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:54.247231 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:54.247729 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0924 18:59:54.248271 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:54.248304 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.249328 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.249335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:54.250624 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:54.250635 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.251408 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.251415 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:54.251674 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:54.252365 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 18:59:54.252391 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 18:59:54.254522 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 18:59:54.257605 (+ 3083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 18:59:54.257613 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 18:59:54.257630 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 18:59:54.258029 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 18:59:54.258414 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 18:59:54.258419 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 18:59:54.258422 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 18:59:54.258592 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 18:59:54.259081 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 18:59:54.259089 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 18:59:54.259421 (+ 332us) client_negotiation.cc:769] Sending connection context
0924 18:59:54.259704 (+ 283us) client_negotiation.cc:240] Negotiation successful
0924 18:59:54.260205 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":368,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260924 18:59:54.262825 1286 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33979
I20260924 18:59:54.263126 1286 heartbeater.cc:464] Registering TS with master...
I20260924 18:59:54.263816 1286 heartbeater.cc:511] Master 127.0.97.62:33979 requested a full tablet report, sending...
I20260924 18:59:54.265180 963 ts_manager.cc:194] Registered new tserver with Master: 86641b1adeed48219b4d4af1f733d9fc (127.0.97.2:46087)
I20260924 18:59:54.266196 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 18:59:54.282279 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33979
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 18:59:54.598081 1291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 18:59:54.598342 1291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 18:59:54.598507 1291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 18:59:54.605741 1291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 18:59:54.605854 1291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 18:59:54.606060 1291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 18:59:54.615460 1291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33979
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 18:59:54.616647 1291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 18:59:54.617939 1291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 18:59:54.630277 1297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 18:59:54.630374 1299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 18:59:54.630333 1296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 18:59:54.631114 1291 server_base.cc:1034] running on GCE node
I20260924 18:59:54.631584 1291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 18:59:54.632351 1291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 18:59:54.633577 1291 hybrid_clock.cc:648] HybridClock initialized: now 1790276394633541 us; error 33 us; skew 500 ppm
I20260924 18:59:54.633895 1291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 18:59:54 dist-test-slave-9fn1 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 18:59:54.636853 1291 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 18:59:54.638218 1291 webserver.cc:460] Webserver started at http://127.0.97.3:41375/ using document root <none> and password file <none>
I20260924 18:59:54.638796 1291 fs_manager.cc:359] Metadata directory not provided
I20260924 18:59:54.638895 1291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 18:59:54.639147 1291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 18:59:54.641840 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-2/data/instance:
uuid: "a86ce7896f884a2db27fd41bcbce6258"
format_stamp: "Formatted at 2026-09-24 18:59:54 on dist-test-slave-9fn1"
server_key: "531c25815df3068c03b988c6a6c55f73"
server_key_iv: "bb6579229697d75558e082c5027883f6"
server_key_version: "encryptionkey@0"
I20260924 18:59:54.642537 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a86ce7896f884a2db27fd41bcbce6258"
format_stamp: "Formatted at 2026-09-24 18:59:54 on dist-test-slave-9fn1"
server_key: "531c25815df3068c03b988c6a6c55f73"
server_key_iv: "bb6579229697d75558e082c5027883f6"
server_key_version: "encryptionkey@0"
I20260924 18:59:54.647367 1291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 18:59:54.650566 1306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 18:59:54.651903 1291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260924 18:59:54.652098 1291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "a86ce7896f884a2db27fd41bcbce6258"
format_stamp: "Formatted at 2026-09-24 18:59:54 on dist-test-slave-9fn1"
server_key: "531c25815df3068c03b988c6a6c55f73"
server_key_iv: "bb6579229697d75558e082c5027883f6"
server_key_version: "encryptionkey@0"
I20260924 18:59:54.652293 1291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 18:59:54.667845 1291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 18:59:54.671703 1291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 18:59:54.671921 1291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 18:59:54.672875 1291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 18:59:54.674268 1291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 18:59:54.674336 1291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 18:59:54.674512 1291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 18:59:54.674559 1291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 18:59:54.705886 1291 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33471
I20260924 18:59:54.705974 1418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33471 every 8 connection(s)
I20260924 18:59:54.707183 1291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 18:59:54.716969 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 1291
I20260924 18:59:54.717149 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276386554690-388-0/minicluster-data/ts-2/wal/instance
I20260924 18:59:54.717618 388 external_mini_cluster.cc:1428] Setting key 79360fab77d92ca62993a2ec8cef7559
Sep 24 18:59:54 dist-test-slave-9fn1 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 18:59:54.726236 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:54.709585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60483 (local address 127.0.97.62:33979)
0924 18:59:54.709748 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:54.709753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:54.710846 (+ 1093us) server_negotiation.cc:407] Connection header received
0924 18:59:54.712750 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:54.712756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:54.712844 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:54.712952 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:54.714495 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.715196 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.716660 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.717138 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.717262 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:54.721074 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:54.721078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:54.721082 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:54.721111 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:54.723763 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:54.724408 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:54.724411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:54.724414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:54.724516 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:54.724874 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:54.724877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:54.724879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:54.725119 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:54.725221 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:54.725868 (+ 647us) server_negotiation.cc:299] Negotiation successful
0924 18:59:54.726021 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 18:59:54.727152 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:54.710022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33979 (local address 127.0.97.3:60483)
0924 18:59:54.710700 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:54.710735 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:54.712483 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:54.713176 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:54.713183 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:54.713821 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0924 18:59:54.714339 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:54.714354 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.715350 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.715355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:54.716521 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:54.716532 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.717358 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.717366 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:54.717666 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:54.718425 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 18:59:54.718454 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 18:59:54.720894 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 18:59:54.723884 (+ 2990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 18:59:54.723890 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 18:59:54.723901 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 18:59:54.724290 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 18:59:54.724602 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 18:59:54.724605 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 18:59:54.724607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 18:59:54.724779 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 18:59:54.725214 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 18:59:54.725220 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 18:59:54.725489 (+ 269us) client_negotiation.cc:769] Sending connection context
0924 18:59:54.725752 (+ 263us) client_negotiation.cc:240] Negotiation successful
0924 18:59:54.726128 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":266,"threads_started":1}
I20260924 18:59:54.728456 1419 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33979
I20260924 18:59:54.728725 1419 heartbeater.cc:464] Registering TS with master...
I20260924 18:59:54.729387 1419 heartbeater.cc:511] Master 127.0.97.62:33979 requested a full tablet report, sending...
I20260924 18:59:54.730829 963 ts_manager.cc:194] Registered new tserver with Master: a86ce7896f884a2db27fd41bcbce6258 (127.0.97.3:33471)
I20260924 18:59:54.731639 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60483
I20260924 18:59:54.732622 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 18:59:54.751017 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:54.739017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59740 (local address 127.0.97.62:33979)
0924 18:59:54.739186 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:54.739190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:54.739708 (+ 518us) server_negotiation.cc:407] Connection header received
0924 18:59:54.739843 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:54.739847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:54.740030 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:54.740137 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:54.741470 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.742216 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.743463 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.743844 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.743930 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:54.746519 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:54.746524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:54.746527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:54.746553 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:54.748905 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:54.749452 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:54.749455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:54.749457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:54.749604 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:54.749898 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:54.749901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:54.749903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:54.750161 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:54.750323 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:54.750631 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 18:59:54.750807 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 18:59:54.756945 963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 18:59:54.763581 963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 18:59:54.785043 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:54.774870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56426 (local address 127.0.97.2:46087)
0924 18:59:54.775515 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:54.775525 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:54.776033 (+ 508us) server_negotiation.cc:407] Connection header received
0924 18:59:54.776387 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:54.776395 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:54.776629 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:54.776835 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 18:59:54.778823 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.779878 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.782692 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.783593 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.783764 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:54.783953 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:54.784562 (+ 609us) server_negotiation.cc:299] Negotiation successful
0924 18:59:54.784711 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":343,"threads_started":1}
I20260924 18:59:54.785264 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:54.776450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33471 (local address 127.0.0.1:60842)
0924 18:59:54.776974 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:54.776989 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:54.777087 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:54.778408 (+ 1321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:54.778415 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:54.778447 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 18:59:54.778933 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:54.778947 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.780936 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.780941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:54.782590 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:54.782627 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.784380 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.784387 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:54.784549 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:54.784582 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 18:59:54.784653 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 18:59:54.784803 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":20,"thread_start_us":600,"threads_started":1}
I20260924 18:59:54.785378 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:54.774807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46087 (local address 127.0.0.1:56426)
0924 18:59:54.775891 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:54.775927 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:54.776112 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:54.777100 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:54.777107 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:54.777136 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 18:59:54.778572 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:54.778587 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.780069 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.780074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:54.782521 (+ 2447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:54.782535 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.784074 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.784082 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:54.784291 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:54.784351 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 18:59:54.784517 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 18:59:54.784697 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":61,"thread_start_us":874,"threads_started":1}
I20260924 18:59:54.785909 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:54.774819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40019 (local address 127.0.0.1:46098)
0924 18:59:54.775898 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:54.775927 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:54.776100 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:54.777318 (+ 1218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:54.777323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:54.777350 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 18:59:54.778537 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:54.778551 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.780363 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.780368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:54.782813 (+ 2445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:54.782829 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.784854 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.784862 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:54.785028 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:54.785061 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 18:59:54.785246 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 18:59:54.785580 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"mutex_wait_us":157,"thread_start_us":865,"threads_started":1}
I20260924 18:59:54.786430 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:54.775483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46098 (local address 127.0.97.1:40019)
0924 18:59:54.776760 (+ 1277us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:54.776770 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:54.776797 (+ 27us) server_negotiation.cc:407] Connection header received
0924 18:59:54.776884 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:54.776891 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:54.777099 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:54.777432 (+ 333us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 18:59:54.778731 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.780216 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.782982 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.784645 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.785729 (+ 1084us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:54.785929 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:54.785989 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 18:59:54.786119 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1114,"thread_start_us":976,"threads_started":1}
I20260924 18:59:54.786521 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:54.776911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60842 (local address 127.0.97.3:33471)
0924 18:59:54.777608 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:54.777617 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:54.777651 (+ 34us) server_negotiation.cc:407] Connection header received
0924 18:59:54.777752 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:54.777758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:54.777953 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:54.778085 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 18:59:54.779121 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.780790 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.782767 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.784156 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.785746 (+ 1590us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:54.785978 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:54.786052 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 18:59:54.786201 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":810,"threads_started":1}
I20260924 18:59:54.789592 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 94eefbbf5ffa4f68bd245cb0fb9ba0d8 (DEFAULT_TABLE table=test-table [id=60da8cc997b34d5b86160d8aaa0736aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 18:59:54.789644 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 94eefbbf5ffa4f68bd245cb0fb9ba0d8 (DEFAULT_TABLE table=test-table [id=60da8cc997b34d5b86160d8aaa0736aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 18:59:54.790014 1354 tablet_service.cc:1467] Processing CreateTablet for tablet 94eefbbf5ffa4f68bd245cb0fb9ba0d8 (DEFAULT_TABLE table=test-table [id=60da8cc997b34d5b86160d8aaa0736aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 18:59:54.790832 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94eefbbf5ffa4f68bd245cb0fb9ba0d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 18:59:54.790948 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94eefbbf5ffa4f68bd245cb0fb9ba0d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 18:59:54.791205 1354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94eefbbf5ffa4f68bd245cb0fb9ba0d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 18:59:54.795120 1152 heartbeater.cc:503] Master 127.0.97.62:33979 was elected leader, sending a full tablet report...
I20260924 18:59:54.803148 1437 tablet_bootstrap.cc:492] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P 63af947c26d24297ba280d4ab33dfd54: Bootstrap starting.
I20260924 18:59:54.806915 1437 tablet_bootstrap.cc:654] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P 63af947c26d24297ba280d4ab33dfd54: Neither blocks nor log segments found. Creating new log.
I20260924 18:59:54.807889 1437 log.cc:824] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P 63af947c26d24297ba280d4ab33dfd54: Log is configured to *not* fsync() on all Append() calls
I20260924 18:59:54.810209 1439 tablet_bootstrap.cc:492] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P a86ce7896f884a2db27fd41bcbce6258: Bootstrap starting.
I20260924 18:59:54.810750 1437 tablet_bootstrap.cc:492] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P 63af947c26d24297ba280d4ab33dfd54: No bootstrap required, opened a new log
I20260924 18:59:54.810998 1437 ts_tablet_manager.cc:1397] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P 63af947c26d24297ba280d4ab33dfd54: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 18:59:54.813421 1439 tablet_bootstrap.cc:654] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P a86ce7896f884a2db27fd41bcbce6258: Neither blocks nor log segments found. Creating new log.
I20260924 18:59:54.814600 1439 log.cc:824] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P a86ce7896f884a2db27fd41bcbce6258: Log is configured to *not* fsync() on all Append() calls
I20260924 18:59:54.817728 1439 tablet_bootstrap.cc:492] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P a86ce7896f884a2db27fd41bcbce6258: No bootstrap required, opened a new log
I20260924 18:59:54.817734 1438 tablet_bootstrap.cc:492] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P 86641b1adeed48219b4d4af1f733d9fc: Bootstrap starting.
I20260924 18:59:54.817961 1439 ts_tablet_manager.cc:1397] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P a86ce7896f884a2db27fd41bcbce6258: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 18:59:54.820158 1438 tablet_bootstrap.cc:654] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P 86641b1adeed48219b4d4af1f733d9fc: Neither blocks nor log segments found. Creating new log.
I20260924 18:59:54.821022 1438 log.cc:824] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P 86641b1adeed48219b4d4af1f733d9fc: Log is configured to *not* fsync() on all Append() calls
I20260924 18:59:54.821170 1437 raft_consensus.cc:348] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P 63af947c26d24297ba280d4ab33dfd54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86641b1adeed48219b4d4af1f733d9fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46087 } } peers { permanent_uuid: "63af947c26d24297ba280d4ab33dfd54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40019 } } peers { permanent_uuid: "a86ce7896f884a2db27fd41bcbce6258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33471 } }
I20260924 18:59:54.821586 1437 raft_consensus.cc:374] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P 63af947c26d24297ba280d4ab33dfd54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 18:59:54.821683 1437 raft_consensus.cc:729] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P 63af947c26d24297ba280d4ab33dfd54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63af947c26d24297ba280d4ab33dfd54, State: Initialized, Role: FOLLOWER
I20260924 18:59:54.822793 1437 consensus_queue.cc:260] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P 63af947c26d24297ba280d4ab33dfd54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86641b1adeed48219b4d4af1f733d9fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46087 } } peers { permanent_uuid: "63af947c26d24297ba280d4ab33dfd54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40019 } } peers { permanent_uuid: "a86ce7896f884a2db27fd41bcbce6258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33471 } }
I20260924 18:59:54.824127 1438 tablet_bootstrap.cc:492] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P 86641b1adeed48219b4d4af1f733d9fc: No bootstrap required, opened a new log
I20260924 18:59:54.824363 1438 ts_tablet_manager.cc:1397] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P 86641b1adeed48219b4d4af1f733d9fc: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.005s
I20260924 18:59:54.825201 1437 ts_tablet_manager.cc:1428] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P 63af947c26d24297ba280d4ab33dfd54: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260924 18:59:54.828301 1439 raft_consensus.cc:348] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P a86ce7896f884a2db27fd41bcbce6258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86641b1adeed48219b4d4af1f733d9fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46087 } } peers { permanent_uuid: "63af947c26d24297ba280d4ab33dfd54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40019 } } peers { permanent_uuid: "a86ce7896f884a2db27fd41bcbce6258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33471 } }
I20260924 18:59:54.828691 1439 raft_consensus.cc:374] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P a86ce7896f884a2db27fd41bcbce6258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 18:59:54.828783 1439 raft_consensus.cc:729] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P a86ce7896f884a2db27fd41bcbce6258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a86ce7896f884a2db27fd41bcbce6258, State: Initialized, Role: FOLLOWER
I20260924 18:59:54.829427 1439 consensus_queue.cc:260] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P a86ce7896f884a2db27fd41bcbce6258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86641b1adeed48219b4d4af1f733d9fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46087 } } peers { permanent_uuid: "63af947c26d24297ba280d4ab33dfd54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40019 } } peers { permanent_uuid: "a86ce7896f884a2db27fd41bcbce6258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33471 } }
I20260924 18:59:54.830382 1419 heartbeater.cc:503] Master 127.0.97.62:33979 was elected leader, sending a full tablet report...
I20260924 18:59:54.831009 1439 ts_tablet_manager.cc:1428] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P a86ce7896f884a2db27fd41bcbce6258: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260924 18:59:54.833892 1438 raft_consensus.cc:348] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P 86641b1adeed48219b4d4af1f733d9fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86641b1adeed48219b4d4af1f733d9fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46087 } } peers { permanent_uuid: "63af947c26d24297ba280d4ab33dfd54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40019 } } peers { permanent_uuid: "a86ce7896f884a2db27fd41bcbce6258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33471 } }
I20260924 18:59:54.834167 1438 raft_consensus.cc:374] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P 86641b1adeed48219b4d4af1f733d9fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 18:59:54.834281 1438 raft_consensus.cc:729] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P 86641b1adeed48219b4d4af1f733d9fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86641b1adeed48219b4d4af1f733d9fc, State: Initialized, Role: FOLLOWER
I20260924 18:59:54.834827 1438 consensus_queue.cc:260] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P 86641b1adeed48219b4d4af1f733d9fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86641b1adeed48219b4d4af1f733d9fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46087 } } peers { permanent_uuid: "63af947c26d24297ba280d4ab33dfd54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40019 } } peers { permanent_uuid: "a86ce7896f884a2db27fd41bcbce6258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33471 } }
I20260924 18:59:54.835753 1286 heartbeater.cc:503] Master 127.0.97.62:33979 was elected leader, sending a full tablet report...
I20260924 18:59:54.836314 1438 ts_tablet_manager.cc:1428] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P 86641b1adeed48219b4d4af1f733d9fc: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260924 18:59:54.890858 1444 raft_consensus.cc:482] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P a86ce7896f884a2db27fd41bcbce6258 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 18:59:54.891172 1444 raft_consensus.cc:504] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P a86ce7896f884a2db27fd41bcbce6258 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86641b1adeed48219b4d4af1f733d9fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46087 } } peers { permanent_uuid: "63af947c26d24297ba280d4ab33dfd54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40019 } } peers { permanent_uuid: "a86ce7896f884a2db27fd41bcbce6258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33471 } }
I20260924 18:59:54.892870 1444 leader_election.cc:288] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P a86ce7896f884a2db27fd41bcbce6258 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86641b1adeed48219b4d4af1f733d9fc (127.0.97.2:46087), 63af947c26d24297ba280d4ab33dfd54 (127.0.97.1:40019)
I20260924 18:59:54.900632 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:54.893240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41653 (local address 127.0.97.2:46087)
0924 18:59:54.893449 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:54.893456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:54.893800 (+ 344us) server_negotiation.cc:407] Connection header received
0924 18:59:54.894021 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:54.894024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:54.894109 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:54.894211 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 18:59:54.894654 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.895700 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.898109 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.899337 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.899899 (+ 562us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:54.900003 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:54.900272 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 18:59:54.900399 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 18:59:54.900676 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:54.893256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46087 (local address 127.0.97.3:41653)
0924 18:59:54.893664 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:54.893693 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:54.893864 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:54.894261 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:54.894265 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:54.894284 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 18:59:54.894511 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:54.894523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.895820 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.895824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:54.897932 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:54.897947 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.899899 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.899903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:54.900100 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:54.900150 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 18:59:54.900241 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 18:59:54.900374 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260}
I20260924 18:59:54.900694 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:54.893369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48341 (local address 127.0.97.1:40019)
0924 18:59:54.893613 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:54.893621 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:54.894119 (+ 498us) server_negotiation.cc:407] Connection header received
0924 18:59:54.894216 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:54.894221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:54.894287 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:54.894377 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 18:59:54.894925 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.895923 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.898254 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.899899 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.900014 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:54.900099 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:54.900383 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 18:59:54.900498 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 18:59:54.901242 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:54.893298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40019 (local address 127.0.97.3:48341)
0924 18:59:54.894025 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:54.894038 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:54.894124 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:54.894399 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:54.894403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:54.894422 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 18:59:54.894781 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:54.894789 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.896057 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.896061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:54.898111 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:54.898123 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.900083 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.900088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:54.900245 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:54.900272 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 18:59:54.900342 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 18:59:54.900428 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"mutex_wait_us":1,"thread_start_us":391,"threads_started":1}
I20260924 18:59:54.901674 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94eefbbf5ffa4f68bd245cb0fb9ba0d8" candidate_uuid: "a86ce7896f884a2db27fd41bcbce6258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86641b1adeed48219b4d4af1f733d9fc" is_pre_election: true
I20260924 18:59:54.902208 1241 raft_consensus.cc:2454] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P 86641b1adeed48219b4d4af1f733d9fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a86ce7896f884a2db27fd41bcbce6258 in term 0.
I20260924 18:59:54.902309 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94eefbbf5ffa4f68bd245cb0fb9ba0d8" candidate_uuid: "a86ce7896f884a2db27fd41bcbce6258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63af947c26d24297ba280d4ab33dfd54" is_pre_election: true
I20260924 18:59:54.902812 1107 raft_consensus.cc:2454] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P 63af947c26d24297ba280d4ab33dfd54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a86ce7896f884a2db27fd41bcbce6258 in term 0.
I20260924 18:59:54.902877 1309 leader_election.cc:302] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P a86ce7896f884a2db27fd41bcbce6258 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86641b1adeed48219b4d4af1f733d9fc, a86ce7896f884a2db27fd41bcbce6258; no voters:
I20260924 18:59:54.903297 1444 raft_consensus.cc:2781] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P a86ce7896f884a2db27fd41bcbce6258 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 18:59:54.903441 1444 raft_consensus.cc:482] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P a86ce7896f884a2db27fd41bcbce6258 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 18:59:54.903548 1444 raft_consensus.cc:3037] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P a86ce7896f884a2db27fd41bcbce6258 [term 0 FOLLOWER]: Advancing to term 1
I20260924 18:59:54.906229 1444 raft_consensus.cc:504] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P a86ce7896f884a2db27fd41bcbce6258 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86641b1adeed48219b4d4af1f733d9fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46087 } } peers { permanent_uuid: "63af947c26d24297ba280d4ab33dfd54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40019 } } peers { permanent_uuid: "a86ce7896f884a2db27fd41bcbce6258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33471 } }
I20260924 18:59:54.906801 1444 leader_election.cc:288] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P a86ce7896f884a2db27fd41bcbce6258 [CANDIDATE]: Term 1 election: Requested vote from peers 86641b1adeed48219b4d4af1f733d9fc (127.0.97.2:46087), 63af947c26d24297ba280d4ab33dfd54 (127.0.97.1:40019)
I20260924 18:59:54.907195 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94eefbbf5ffa4f68bd245cb0fb9ba0d8" candidate_uuid: "a86ce7896f884a2db27fd41bcbce6258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86641b1adeed48219b4d4af1f733d9fc"
I20260924 18:59:54.907269 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94eefbbf5ffa4f68bd245cb0fb9ba0d8" candidate_uuid: "a86ce7896f884a2db27fd41bcbce6258" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63af947c26d24297ba280d4ab33dfd54"
I20260924 18:59:54.907400 1241 raft_consensus.cc:3037] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P 86641b1adeed48219b4d4af1f733d9fc [term 0 FOLLOWER]: Advancing to term 1
I20260924 18:59:54.907469 1107 raft_consensus.cc:3037] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P 63af947c26d24297ba280d4ab33dfd54 [term 0 FOLLOWER]: Advancing to term 1
I20260924 18:59:54.909593 1241 raft_consensus.cc:2454] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P 86641b1adeed48219b4d4af1f733d9fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a86ce7896f884a2db27fd41bcbce6258 in term 1.
I20260924 18:59:54.909626 1107 raft_consensus.cc:2454] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P 63af947c26d24297ba280d4ab33dfd54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a86ce7896f884a2db27fd41bcbce6258 in term 1.
I20260924 18:59:54.909999 1309 leader_election.cc:302] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P a86ce7896f884a2db27fd41bcbce6258 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86641b1adeed48219b4d4af1f733d9fc, a86ce7896f884a2db27fd41bcbce6258; no voters:
I20260924 18:59:54.910333 1444 raft_consensus.cc:2781] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P a86ce7896f884a2db27fd41bcbce6258 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 18:59:54.910790 1444 raft_consensus.cc:686] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P a86ce7896f884a2db27fd41bcbce6258 [term 1 LEADER]: Becoming Leader. State: Replica: a86ce7896f884a2db27fd41bcbce6258, State: Running, Role: LEADER
I20260924 18:59:54.911324 1444 consensus_queue.cc:237] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P a86ce7896f884a2db27fd41bcbce6258 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86641b1adeed48219b4d4af1f733d9fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46087 } } peers { permanent_uuid: "63af947c26d24297ba280d4ab33dfd54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40019 } } peers { permanent_uuid: "a86ce7896f884a2db27fd41bcbce6258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33471 } }
I20260924 18:59:54.915846 962 catalog_manager.cc:5495] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P a86ce7896f884a2db27fd41bcbce6258 reported cstate change: term changed from 0 to 1, leader changed from <none> to a86ce7896f884a2db27fd41bcbce6258 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "a86ce7896f884a2db27fd41bcbce6258" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86641b1adeed48219b4d4af1f733d9fc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63af947c26d24297ba280d4ab33dfd54" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a86ce7896f884a2db27fd41bcbce6258" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33471 } health_report { overall_health: HEALTHY } } }
Sep 24 18:59:54 dist-test-slave-9fn1 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 18:59:54.944132 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:54.929871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46104 (local address 127.0.97.1:40019)
0924 18:59:54.930025 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:54.930029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:54.930510 (+ 481us) server_negotiation.cc:407] Connection header received
0924 18:59:54.930722 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:54.930728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:54.930810 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:54.930915 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:54.932404 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.933128 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.934498 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.934925 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.935054 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:54.938729 (+ 3675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:54.938741 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:54.938756 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:54.938784 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:54.941948 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:54.942507 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:54.942514 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:54.942516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:54.942624 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:54.942909 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:54.942912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:54.942914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:54.943216 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:54.943440 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:54.943610 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 18:59:54.943748 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 18:59:54.958747 1420 tablet.cc:2369] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P a86ce7896f884a2db27fd41bcbce6258: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 18:59:54 dist-test-slave-9fn1 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276394, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 18:59:54 dist-test-slave-9fn1 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276394, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 18:59:54.988910 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:54.975978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46106 (local address 127.0.97.1:40019)
0924 18:59:54.976215 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:54.976221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:54.976445 (+ 224us) server_negotiation.cc:407] Connection header received
0924 18:59:54.976581 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:54.976585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:54.976645 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:54.976728 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:54.977980 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.978684 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.979853 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:54.980228 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:54.980316 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:54.983818 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:54.983823 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:54.983827 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:54.983859 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:54.986809 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:54.987353 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:54.987357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:54.987359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:54.987474 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:54.987770 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:54.987773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:54.987775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:54.988006 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:54.988189 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:54.988394 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 18:59:54.988541 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 18:59:54.992785 1287 tablet.cc:2369] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P 86641b1adeed48219b4d4af1f733d9fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 18:59:55 dist-test-slave-9fn1 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260924 18:59:55.013772 1153 tablet.cc:2369] T 94eefbbf5ffa4f68bd245cb0fb9ba0d8 P 63af947c26d24297ba280d4ab33dfd54: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 18:59:55 dist-test-slave-9fn1 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 18:59:55.030995 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:55.017203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46110 (local address 127.0.97.1:40019)
0924 18:59:55.017422 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:55.017430 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:55.017765 (+ 335us) server_negotiation.cc:407] Connection header received
0924 18:59:55.017973 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:55.017979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:55.018048 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:55.018172 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:55.019455 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:55.020112 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:55.021297 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:55.021773 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:55.021910 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:55.025403 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:55.025408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:55.025411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:55.025440 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:55.028049 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:55.028686 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:55.028692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:55.028696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:55.028816 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:55.029173 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:55.029179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:55.029183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:55.029450 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:55.029764 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:55.030052 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 18:59:55.030269 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 18:59:55.032626 1067 tablet_service.cc:2908] Rejecting scan request for tablet 94eefbbf5ffa4f68bd245cb0fb9ba0d8: Uninitialized: clean time has not yet been initialized
I20260924 18:59:55.036399 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 1024
I20260924 18:59:55.047515 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:55.037059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46120 (local address 127.0.97.1:40019)
0924 18:59:55.037204 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:55.037208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:55.037638 (+ 430us) server_negotiation.cc:407] Connection header received
0924 18:59:55.037756 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:55.037760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:55.037811 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:55.037885 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:55.038916 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:55.039556 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:55.040673 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:55.041167 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:55.041267 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:55.043287 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:55.043291 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:55.043294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:55.043318 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:55.045484 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:55.046070 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:55.046074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:55.046076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:55.046186 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:55.046466 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:55.046469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:55.046471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:55.046704 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:55.046951 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:55.047147 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 18:59:55.047302 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 18:59:55.048225 1147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 18:59:55.134703 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 1024
I20260924 18:59:55.151865 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 1158
Sep 24 18:59:55 dist-test-slave-9fn1 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 18:59:55.165591 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:55.152722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56432 (local address 127.0.97.2:46087)
0924 18:59:55.152901 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:55.152905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:55.152922 (+ 17us) server_negotiation.cc:407] Connection header received
0924 18:59:55.152973 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:55.152977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:55.153032 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:55.153111 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:55.154235 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:55.154935 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:55.156225 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:55.156659 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:55.156753 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:55.160303 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:55.160317 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:55.160333 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:55.160365 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:55.163156 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:55.163711 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:55.163717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:55.163720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:55.163858 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:55.164181 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:55.164184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:55.164186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:55.164503 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:55.164786 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:55.164909 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 18:59:55.165145 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 18:59:55.166237 1281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 18:59:55.262228 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 1158
I20260924 18:59:55.278524 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 1291
Sep 24 18:59:55 dist-test-slave-9fn1 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 18:59:55.292335 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:55.279306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60844 (local address 127.0.97.3:33471)
0924 18:59:55.279464 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:55.279468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:55.279486 (+ 18us) server_negotiation.cc:407] Connection header received
0924 18:59:55.279587 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:55.279591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:55.279658 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:55.279740 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:55.280921 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:55.281781 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:55.283043 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:55.283413 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:55.283500 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:55.287664 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:55.287676 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:55.287691 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:55.287723 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:55.290205 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:55.290776 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:55.290782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:55.290784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:55.290898 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:55.291220 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:55.291223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:55.291225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:55.291526 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:55.291744 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:55.291929 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 18:59:55.292106 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 18:59:55.292949 1414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 18:59:55.379527 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 1291
I20260924 18:59:55.395411 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 931
I20260924 18:59:55.396648 993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 18:59:55.499827 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 931
2026-09-24T18:59:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3934 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 18:59:55 dist-test-slave-9fn1 krb5kdc[1466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 18:59:55 dist-test-slave-9fn1 krb5kdc[1466](info): set up 2 sockets
Sep 24 18:59:55 dist-test-slave-9fn1 krb5kdc[1466](info): commencing operation
krb5kdc: starting...
W20260924 18:59:56.565521 388 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.010s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 18:59:56 dist-test-slave-9fn1 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T18:59:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T18:59:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 18:59:56.740387 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41805
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 18:59:57.053560 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 18:59:57.053884 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 18:59:57.054011 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 18:59:57.060956 1479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 18:59:57.061048 1479 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 18:59:57.061087 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 18:59:57.061148 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 18:59:57.061197 1479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 18:59:57.061249 1479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 18:59:57.070863 1479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41805
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 18:59:57.072019 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 18:59:57.073213 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 18:59:57.084596 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 18:59:57.084540 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 18:59:57.084734 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 18:59:57.085179 1479 server_base.cc:1034] running on GCE node
I20260924 18:59:57.085971 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 18:59:57.087193 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 18:59:57.088403 1479 hybrid_clock.cc:648] HybridClock initialized: now 1790276397088340 us; error 36 us; skew 500 ppm
I20260924 18:59:57.088670 1479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 18:59:57 dist-test-slave-9fn1 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 18:59:57.095654 1479 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 18:59:57.096951 1479 webserver.cc:460] Webserver started at http://127.0.97.62:45631/ using document root <none> and password file <none>
I20260924 18:59:57.097637 1479 fs_manager.cc:359] Metadata directory not provided
I20260924 18:59:57.097726 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 18:59:57.097980 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 18:59:57.100649 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "24931820b37e4f298ea207374672357e"
format_stamp: "Formatted at 2026-09-24 18:59:57 on dist-test-slave-9fn1"
server_key: "23ddfc49c22e1c5e5ad5e284fb3189b8"
server_key_iv: "95487fcce7949610084031446e798c29"
server_key_version: "encryptionkey@0"
I20260924 18:59:57.101361 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "24931820b37e4f298ea207374672357e"
format_stamp: "Formatted at 2026-09-24 18:59:57 on dist-test-slave-9fn1"
server_key: "23ddfc49c22e1c5e5ad5e284fb3189b8"
server_key_iv: "95487fcce7949610084031446e798c29"
server_key_version: "encryptionkey@0"
I20260924 18:59:57.106052 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 18:59:57.109542 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 18:59:57.110721 1479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 18:59:57.110941 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "24931820b37e4f298ea207374672357e"
format_stamp: "Formatted at 2026-09-24 18:59:57 on dist-test-slave-9fn1"
server_key: "23ddfc49c22e1c5e5ad5e284fb3189b8"
server_key_iv: "95487fcce7949610084031446e798c29"
server_key_version: "encryptionkey@0"
I20260924 18:59:57.111152 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 18:59:57.128576 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 18:59:57.132256 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 18:59:57.132526 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 18:59:57.152957 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41805
I20260924 18:59:57.152978 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41805 every 8 connection(s)
I20260924 18:59:57.154309 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 18:59:57.155735 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 1479
I20260924 18:59:57.155957 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 18:59:57.156610 388 external_mini_cluster.cc:1428] Setting key 09f7d663e804367470ffc8aed11ba392
I20260924 18:59:57.158401 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 18:59:57 dist-test-slave-9fn1 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276396, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 18:59:57.171360 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24931820b37e4f298ea207374672357e: Bootstrap starting.
I20260924 18:59:57.175230 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24931820b37e4f298ea207374672357e: Neither blocks nor log segments found. Creating new log.
I20260924 18:59:57.176401 1556 log.cc:824] T 00000000000000000000000000000000 P 24931820b37e4f298ea207374672357e: Log is configured to *not* fsync() on all Append() calls
I20260924 18:59:57.179193 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24931820b37e4f298ea207374672357e: No bootstrap required, opened a new log
I20260924 18:59:57.179679 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:57.158547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39372 (local address 127.0.97.62:41805)
0924 18:59:57.159306 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:57.159320 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:57.159367 (+ 47us) server_negotiation.cc:407] Connection header received
0924 18:59:57.161403 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:57.161436 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:57.161957 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:57.162380 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:57.163588 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:57.165625 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:57.166800 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:57.167781 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:57.167963 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:57.171930 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:57.171940 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:57.171957 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:57.171996 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:57.175796 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:57.176411 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:57.176429 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:57.176437 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:57.176632 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:57.176991 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:57.176997 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:57.177000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:57.177556 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:57.177854 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:57.178306 (+ 452us) server_negotiation.cc:299] Negotiation successful
0924 18:59:57.178713 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":313,"threads_started":1}
I20260924 18:59:57.186352 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24931820b37e4f298ea207374672357e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24931820b37e4f298ea207374672357e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41805 } }
I20260924 18:59:57.186672 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24931820b37e4f298ea207374672357e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 18:59:57.186769 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24931820b37e4f298ea207374672357e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24931820b37e4f298ea207374672357e, State: Initialized, Role: FOLLOWER
I20260924 18:59:57.187364 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24931820b37e4f298ea207374672357e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24931820b37e4f298ea207374672357e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41805 } }
I20260924 18:59:57.187592 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24931820b37e4f298ea207374672357e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 18:59:57.187721 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24931820b37e4f298ea207374672357e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 18:59:57.187906 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24931820b37e4f298ea207374672357e [term 0 FOLLOWER]: Advancing to term 1
I20260924 18:59:57.189929 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24931820b37e4f298ea207374672357e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24931820b37e4f298ea207374672357e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41805 } }
I20260924 18:59:57.190383 1556 leader_election.cc:302] T 00000000000000000000000000000000 P 24931820b37e4f298ea207374672357e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24931820b37e4f298ea207374672357e; no voters:
I20260924 18:59:57.190932 1556 leader_election.cc:288] T 00000000000000000000000000000000 P 24931820b37e4f298ea207374672357e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 18:59:57.191112 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24931820b37e4f298ea207374672357e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 18:59:57.192224 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24931820b37e4f298ea207374672357e [term 1 LEADER]: Becoming Leader. State: Replica: 24931820b37e4f298ea207374672357e, State: Running, Role: LEADER
I20260924 18:59:57.192695 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24931820b37e4f298ea207374672357e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24931820b37e4f298ea207374672357e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41805 } }
I20260924 18:59:57.192955 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24931820b37e4f298ea207374672357e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 18:59:57.195448 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24931820b37e4f298ea207374672357e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24931820b37e4f298ea207374672357e. Latest consensus state: current_term: 1 leader_uuid: "24931820b37e4f298ea207374672357e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24931820b37e4f298ea207374672357e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41805 } } }
I20260924 18:59:57.195868 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24931820b37e4f298ea207374672357e [sys.catalog]: This master's current role is: LEADER
I20260924 18:59:57.195824 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24931820b37e4f298ea207374672357e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24931820b37e4f298ea207374672357e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24931820b37e4f298ea207374672357e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41805 } } }
I20260924 18:59:57.196065 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24931820b37e4f298ea207374672357e [sys.catalog]: This master's current role is: LEADER
I20260924 18:59:57.204304 1570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 18:59:57.210153 1570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 18:59:57.220232 1570 catalog_manager.cc:1347] Generated new cluster ID: a6a12e16f0ae46f99fe8446196b0115c
I20260924 18:59:57.220340 1570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 18:59:57.269735 1570 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 18:59:57.271005 1570 catalog_manager.cc:1504] Loading token signing keys...
I20260924 18:59:57.282588 1570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24931820b37e4f298ea207374672357e: Generated new TSK 0
I20260924 18:59:57.283550 1570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 18:59:57.335608 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41805
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 18:59:57.650051 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 18:59:57.650375 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 18:59:57.650547 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 18:59:57.657614 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 18:59:57.657757 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 18:59:57.657927 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 18:59:57.658064 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 18:59:57.667438 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:41805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 18:59:57.668666 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 18:59:57.669962 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 18:59:57.682822 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 18:59:57.682857 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 18:59:57.682895 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 18:59:57.683640 1582 server_base.cc:1034] running on GCE node
I20260924 18:59:57.684237 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 18:59:57.685083 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 18:59:57.686317 1582 hybrid_clock.cc:648] HybridClock initialized: now 1790276397686247 us; error 50 us; skew 500 ppm
I20260924 18:59:57.686607 1582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 18:59:57 dist-test-slave-9fn1 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 18:59:57.689457 1582 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 18:59:57.690749 1582 webserver.cc:460] Webserver started at http://127.0.97.1:34839/ using document root <none> and password file <none>
I20260924 18:59:57.691406 1582 fs_manager.cc:359] Metadata directory not provided
I20260924 18:59:57.691530 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 18:59:57.691819 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 18:59:57.694751 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-0/data/instance:
uuid: "afb2353e1d634837966bb862c6e9753a"
format_stamp: "Formatted at 2026-09-24 18:59:57 on dist-test-slave-9fn1"
server_key: "082d7db1e88fb7b7ae041bd2920935ea"
server_key_iv: "565a68b16acd8db0cd00e7bcf48bbb1d"
server_key_version: "encryptionkey@0"
I20260924 18:59:57.695531 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-0/wal/instance:
uuid: "afb2353e1d634837966bb862c6e9753a"
format_stamp: "Formatted at 2026-09-24 18:59:57 on dist-test-slave-9fn1"
server_key: "082d7db1e88fb7b7ae041bd2920935ea"
server_key_iv: "565a68b16acd8db0cd00e7bcf48bbb1d"
server_key_version: "encryptionkey@0"
I20260924 18:59:57.700273 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 18:59:57.703619 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 18:59:57.704757 1582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 18:59:57.704919 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "afb2353e1d634837966bb862c6e9753a"
format_stamp: "Formatted at 2026-09-24 18:59:57 on dist-test-slave-9fn1"
server_key: "082d7db1e88fb7b7ae041bd2920935ea"
server_key_iv: "565a68b16acd8db0cd00e7bcf48bbb1d"
server_key_version: "encryptionkey@0"
I20260924 18:59:57.705111 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 18:59:57.723820 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 18:59:57.727671 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 18:59:57.727936 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 18:59:57.730685 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 18:59:57.730787 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 18:59:57.730931 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 18:59:57.731014 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 18:59:57 dist-test-slave-9fn1 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 18:59:57.754123 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:57.731138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60251 (local address 127.0.97.62:41805)
0924 18:59:57.731994 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:57.731999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:57.732875 (+ 876us) server_negotiation.cc:407] Connection header received
0924 18:59:57.734961 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:57.734967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:57.735050 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:57.735181 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:57.737627 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:57.738689 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:57.741353 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:57.741980 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:57.742123 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:57.747335 (+ 5212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:57.747340 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:57.747344 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:57.747374 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:57.749985 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:57.751113 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:57.751120 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:57.751127 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:57.751284 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:57.751823 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:57.751829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:57.751832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:57.752232 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:57.752420 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:57.753483 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0924 18:59:57.753784 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":576,"threads_started":1}
I20260924 18:59:57.755901 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:57.731800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41805 (local address 127.0.97.1:60251)
0924 18:59:57.732708 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:57.732744 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:57.734672 (+ 1928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:57.735431 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:57.735439 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:57.736341 (+ 902us) client_negotiation.cc:189] Negotiated authn=SASL
0924 18:59:57.737395 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:57.737418 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:57.738881 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:57.738885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:57.741197 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:57.741209 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:57.742475 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:57.742486 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:57.742814 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:57.743671 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 18:59:57.743692 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 18:59:57.747090 (+ 3398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 18:59:57.750139 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 18:59:57.750157 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 18:59:57.750182 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 18:59:57.750955 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 18:59:57.751417 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 18:59:57.751424 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 18:59:57.751433 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 18:59:57.751699 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 18:59:57.752387 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 18:59:57.752399 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 18:59:57.752978 (+ 579us) client_negotiation.cc:769] Sending connection context
0924 18:59:57.753364 (+ 386us) client_negotiation.cc:240] Negotiation successful
0924 18:59:57.754082 (+ 718us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":429,"threads_started":1}
I20260924 18:59:57.774966 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:57.761408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44111 (local address 127.0.97.62:41805)
0924 18:59:57.761610 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:57.761616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:57.762408 (+ 792us) server_negotiation.cc:407] Connection header received
0924 18:59:57.762671 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:57.762676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:57.762745 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:57.762842 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:57.763713 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:57.764752 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:57.766293 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:57.766889 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:57.767033 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:57.768766 (+ 1733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:57.768772 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:57.768777 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:57.768817 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:57.771826 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:57.772752 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:57.772759 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:57.772765 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:57.772909 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:57.773257 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:57.773264 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:57.773268 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:57.773595 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:57.773738 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:57.774395 (+ 657us) server_negotiation.cc:299] Negotiation successful
0924 18:59:57.774659 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 18:59:57.775805 1665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:57.761434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41805 (local address 127.0.97.1:44111)
0924 18:59:57.762298 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:57.762319 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:57.762471 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:57.762865 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:57.762870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:57.763299 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0924 18:59:57.763576 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:57.763585 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:57.764874 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:57.764881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:57.766141 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:57.766156 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:57.767106 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:57.767117 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:57.767276 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:57.767932 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 18:59:57.767947 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 18:59:57.768603 (+ 656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 18:59:57.771962 (+ 3359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 18:59:57.771970 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 18:59:57.771974 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 18:59:57.772632 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 18:59:57.773026 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 18:59:57.773031 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 18:59:57.773035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 18:59:57.773155 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 18:59:57.773704 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 18:59:57.773708 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 18:59:57.773978 (+ 270us) client_negotiation.cc:769] Sending connection context
0924 18:59:57.775170 (+ 1192us) client_negotiation.cc:240] Negotiation successful
0924 18:59:57.775485 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":485,"threads_started":1}
I20260924 18:59:57.784749 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42503
I20260924 18:59:57.784778 1717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42503 every 8 connection(s)
I20260924 18:59:57.785976 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 18:59:57.787837 1718 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41805
I20260924 18:59:57.788064 1718 heartbeater.cc:464] Registering TS with master...
I20260924 18:59:57.788686 1718 heartbeater.cc:511] Master 127.0.97.62:41805 requested a full tablet report, sending...
I20260924 18:59:57.790529 1511 ts_manager.cc:194] Registered new tserver with Master: afb2353e1d634837966bb862c6e9753a (127.0.97.1:42503)
I20260924 18:59:57.792075 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60251
I20260924 18:59:57.793540 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 1582
I20260924 18:59:57.793744 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-0/wal/instance
I20260924 18:59:57.794478 388 external_mini_cluster.cc:1428] Setting key 2207579bc2a59d9d842e31f8b8231fc0
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 18:59:57.831136 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41805
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 18:59:58.138689 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 18:59:58.139042 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 18:59:58.139202 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 18:59:58.146063 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 18:59:58.146200 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 18:59:58.146364 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 18:59:58.146502 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 18:59:58.158171 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:41805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 18:59:58.159579 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 18:59:58.161087 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 18:59:58.175392 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 18:59:58.175324 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 18:59:58.175364 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 18:59:58.176275 1722 server_base.cc:1034] running on GCE node
I20260924 18:59:58.176896 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 18:59:58.177734 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 18:59:58.178934 1722 hybrid_clock.cc:648] HybridClock initialized: now 1790276398178855 us; error 56 us; skew 500 ppm
I20260924 18:59:58.179169 1722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 18:59:58 dist-test-slave-9fn1 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 18:59:58.189170 1722 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 18:59:58.190609 1722 webserver.cc:460] Webserver started at http://127.0.97.2:42681/ using document root <none> and password file <none>
I20260924 18:59:58.191290 1722 fs_manager.cc:359] Metadata directory not provided
I20260924 18:59:58.191408 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 18:59:58.191735 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 18:59:58.194649 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-1/data/instance:
uuid: "a402da4abc2a40309caa3ab7663c093a"
format_stamp: "Formatted at 2026-09-24 18:59:58 on dist-test-slave-9fn1"
server_key: "72b99009d5e00de8161b45d07a905883"
server_key_iv: "8e93d4038ec64e34ecaee30469ee511a"
server_key_version: "encryptionkey@0"
I20260924 18:59:58.195430 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a402da4abc2a40309caa3ab7663c093a"
format_stamp: "Formatted at 2026-09-24 18:59:58 on dist-test-slave-9fn1"
server_key: "72b99009d5e00de8161b45d07a905883"
server_key_iv: "8e93d4038ec64e34ecaee30469ee511a"
server_key_version: "encryptionkey@0"
I20260924 18:59:58.200443 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 18:59:58.203766 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 18:59:58.205406 1722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260924 18:59:58.205591 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "a402da4abc2a40309caa3ab7663c093a"
format_stamp: "Formatted at 2026-09-24 18:59:58 on dist-test-slave-9fn1"
server_key: "72b99009d5e00de8161b45d07a905883"
server_key_iv: "8e93d4038ec64e34ecaee30469ee511a"
server_key_version: "encryptionkey@0"
I20260924 18:59:58.205880 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 18:59:58.221742 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 18:59:58.225271 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 18:59:58.225549 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 18:59:58.228001 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 18:59:58.228124 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 18:59:58.228283 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 18:59:58.228363 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 18:59:58 dist-test-slave-9fn1 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 18:59:58.249679 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:58.228343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46043 (local address 127.0.97.62:41805)
0924 18:59:58.228536 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:58.228544 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:58.230021 (+ 1477us) server_negotiation.cc:407] Connection header received
0924 18:59:58.232894 (+ 2873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:58.232900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:58.232991 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:58.233115 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:58.235272 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.236613 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.238914 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.239540 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.239684 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:58.243277 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:58.243283 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:58.243287 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:58.243335 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:58.246711 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:58.247411 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:58.247418 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:58.247426 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:58.247559 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:58.247984 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:58.247990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:58.247994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:58.248248 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:58.248401 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:58.249077 (+ 676us) server_negotiation.cc:299] Negotiation successful
0924 18:59:58.249283 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 18:59:58.250774 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:58.228957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41805 (local address 127.0.97.2:46043)
0924 18:59:58.229843 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:58.229891 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:58.232557 (+ 2666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:58.233403 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:58.233417 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:58.234150 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0924 18:59:58.235060 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:58.235082 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.236769 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.236783 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:58.238767 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:58.238778 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.239788 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.239796 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:58.240000 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:58.240638 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 18:59:58.240658 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 18:59:58.243046 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 18:59:58.246857 (+ 3811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 18:59:58.246865 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 18:59:58.246881 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 18:59:58.247283 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 18:59:58.247669 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 18:59:58.247675 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 18:59:58.247679 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 18:59:58.247867 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 18:59:58.248375 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 18:59:58.248382 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 18:59:58.248706 (+ 324us) client_negotiation.cc:769] Sending connection context
0924 18:59:58.248955 (+ 249us) client_negotiation.cc:240] Negotiation successful
0924 18:59:58.249442 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":364,"threads_started":1}
I20260924 18:59:58.273610 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46661
I20260924 18:59:58.273664 1856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46661 every 8 connection(s)
I20260924 18:59:58.274466 1811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:58.257913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41805 (local address 127.0.97.2:37045)
0924 18:59:58.258570 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:58.258636 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:58.258761 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:58.259100 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:58.259105 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:58.259949 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0924 18:59:58.260428 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:58.260442 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.262107 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.262114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:58.264255 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:58.264278 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.265254 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.265259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:58.265374 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:58.266050 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 18:59:58.266080 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 18:59:58.267375 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 18:59:58.271661 (+ 4286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 18:59:58.271668 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 18:59:58.271671 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 18:59:58.272029 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 18:59:58.272436 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 18:59:58.272440 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 18:59:58.272444 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 18:59:58.272584 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 18:59:58.273337 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 18:59:58.273344 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 18:59:58.273736 (+ 392us) client_negotiation.cc:769] Sending connection context
0924 18:59:58.273927 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 18:59:58.274145 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":605,"threads_started":1}
I20260924 18:59:58.274726 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:58.258448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37045 (local address 127.0.97.62:41805)
0924 18:59:58.258641 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:58.258647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:58.258694 (+ 47us) server_negotiation.cc:407] Connection header received
0924 18:59:58.258888 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:58.258895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:58.258994 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:58.259084 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:58.260640 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.261916 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.264461 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.265091 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.265222 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:58.267622 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:58.267628 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:58.267633 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:58.267670 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:58.271548 (+ 3878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:58.272150 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:58.272158 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:58.272161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:58.272320 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:58.272724 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:58.272731 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:58.272737 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:58.273178 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:58.273342 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:58.274013 (+ 671us) server_negotiation.cc:299] Negotiation successful
0924 18:59:58.274249 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 18:59:58.275207 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 18:59:58.277202 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 1722
I20260924 18:59:58.277355 1857 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41805
I20260924 18:59:58.277429 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-1/wal/instance
I20260924 18:59:58.277906 388 external_mini_cluster.cc:1428] Setting key 5893ba23ffca27c23c316ffa50ba72a9
I20260924 18:59:58.277983 1857 heartbeater.cc:464] Registering TS with master...
I20260924 18:59:58.279912 1857 heartbeater.cc:511] Master 127.0.97.62:41805 requested a full tablet report, sending...
I20260924 18:59:58.281126 1511 ts_manager.cc:194] Registered new tserver with Master: a402da4abc2a40309caa3ab7663c093a (127.0.97.2:46661)
I20260924 18:59:58.281960 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46043
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 18:59:58.314440 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41805
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 18:59:58.627497 1861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 18:59:58.627753 1861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 18:59:58.627909 1861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 18:59:58.634965 1861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 18:59:58.635108 1861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 18:59:58.635262 1861 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 18:59:58.635401 1861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 18:59:58.645326 1861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:41805
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 18:59:58.646615 1861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 18:59:58.647913 1861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 18:59:58.660811 1866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 18:59:58.660848 1869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 18:59:58.660822 1867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 18:59:58.661782 1861 server_base.cc:1034] running on GCE node
I20260924 18:59:58.662350 1861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 18:59:58.663214 1861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 18:59:58.664434 1861 hybrid_clock.cc:648] HybridClock initialized: now 1790276398664344 us; error 75 us; skew 500 ppm
I20260924 18:59:58.664690 1861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 18:59:58 dist-test-slave-9fn1 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 18:59:58.667765 1861 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 18:59:58.669044 1861 webserver.cc:460] Webserver started at http://127.0.97.3:39901/ using document root <none> and password file <none>
I20260924 18:59:58.669746 1861 fs_manager.cc:359] Metadata directory not provided
I20260924 18:59:58.669831 1861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 18:59:58.670085 1861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 18:59:58.672750 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-2/data/instance:
uuid: "bd36d282dbbe4a58a9895b149f77ace9"
format_stamp: "Formatted at 2026-09-24 18:59:58 on dist-test-slave-9fn1"
server_key: "9a751b2f6d9ad35adff761bcef152362"
server_key_iv: "02a081ed9e34aa22efcee896ac5dce61"
server_key_version: "encryptionkey@0"
I20260924 18:59:58.673453 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-2/wal/instance:
uuid: "bd36d282dbbe4a58a9895b149f77ace9"
format_stamp: "Formatted at 2026-09-24 18:59:58 on dist-test-slave-9fn1"
server_key: "9a751b2f6d9ad35adff761bcef152362"
server_key_iv: "02a081ed9e34aa22efcee896ac5dce61"
server_key_version: "encryptionkey@0"
I20260924 18:59:58.678262 1861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 18:59:58.681385 1876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 18:59:58.682756 1861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 18:59:58.682929 1861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "bd36d282dbbe4a58a9895b149f77ace9"
format_stamp: "Formatted at 2026-09-24 18:59:58 on dist-test-slave-9fn1"
server_key: "9a751b2f6d9ad35adff761bcef152362"
server_key_iv: "02a081ed9e34aa22efcee896ac5dce61"
server_key_version: "encryptionkey@0"
I20260924 18:59:58.683138 1861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 18:59:58.705276 1861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 18:59:58.709648 1861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 18:59:58.709910 1861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 18:59:58.712481 1861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 18:59:58.712589 1861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 18:59:58.712713 1861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 18:59:58.712782 1861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 18:59:58 dist-test-slave-9fn1 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 18:59:58.735630 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:58.713084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33801 (local address 127.0.97.62:41805)
0924 18:59:58.713280 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:58.713287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:58.714879 (+ 1592us) server_negotiation.cc:407] Connection header received
0924 18:59:58.716895 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:58.716900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:58.716988 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:58.717094 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:58.719141 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.720125 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.722860 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.723428 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.723555 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:58.728363 (+ 4808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:58.728370 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:58.728376 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:58.728415 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:58.731852 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:58.732808 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:58.732815 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:58.732820 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:58.732989 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:58.733596 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:58.733601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:58.733604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:58.734027 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:58.734158 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:58.735061 (+ 903us) server_negotiation.cc:299] Negotiation successful
0924 18:59:58.735282 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 18:59:58.736865 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:58.713848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41805 (local address 127.0.97.3:33801)
0924 18:59:58.714727 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:58.714771 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:58.716599 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:58.717323 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:58.717330 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:58.718072 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0924 18:59:58.718913 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:58.718940 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.720293 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.720298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:58.722719 (+ 2421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:58.722737 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.723648 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.723655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:58.723964 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:58.724743 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 18:59:58.724783 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 18:59:58.728101 (+ 3318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 18:59:58.731985 (+ 3884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 18:59:58.731993 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 18:59:58.732011 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 18:59:58.732619 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 18:59:58.733112 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 18:59:58.733117 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 18:59:58.733120 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 18:59:58.733441 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 18:59:58.734154 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 18:59:58.734160 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 18:59:58.734647 (+ 487us) client_negotiation.cc:769] Sending connection context
0924 18:59:58.734949 (+ 302us) client_negotiation.cc:240] Negotiation successful
0924 18:59:58.735487 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":451,"threads_started":1}
I20260924 18:59:58.752228 1861 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42001
I20260924 18:59:58.752270 1995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42001 every 8 connection(s)
I20260924 18:59:58.753669 1861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 18:59:58.756047 1996 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41805
I20260924 18:59:58.756317 1996 heartbeater.cc:464] Registering TS with master...
I20260924 18:59:58.757047 1996 heartbeater.cc:511] Master 127.0.97.62:41805 requested a full tablet report, sending...
I20260924 18:59:58.758512 1511 ts_manager.cc:194] Registered new tserver with Master: bd36d282dbbe4a58a9895b149f77ace9 (127.0.97.3:42001)
I20260924 18:59:58.759325 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 1861
I20260924 18:59:58.759496 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-2/wal/instance
I20260924 18:59:58.759603 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33801
I20260924 18:59:58.760012 388 external_mini_cluster.cc:1428] Setting key b05f310547b0f970f5dd4b96c53f0948
I20260924 18:59:58.761898 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:58.748749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60889 (local address 127.0.97.62:41805)
0924 18:59:58.749065 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:58.749073 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:58.749217 (+ 144us) server_negotiation.cc:407] Connection header received
0924 18:59:58.749548 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:58.749555 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:58.749641 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:58.749758 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:58.750750 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.751713 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.753215 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.753743 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.753876 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:58.755889 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:58.755897 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:58.755902 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:58.755940 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:58.759070 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:58.759688 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:58.759694 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:58.759698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:58.759835 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:58.760250 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:58.760255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:58.760259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:58.760614 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:58.760762 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:58.761286 (+ 524us) server_negotiation.cc:299] Negotiation successful
0924 18:59:58.761488 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260924 18:59:58.761937 1988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:58.748263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41805 (local address 127.0.97.3:60889)
0924 18:59:58.749072 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:58.749110 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:58.749290 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:58.749796 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:58.749800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:58.750312 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0924 18:59:58.750594 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:58.750603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.751853 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.751859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:58.753081 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:58.753093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.753923 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.753930 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:58.754158 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:58.754947 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 18:59:58.754964 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 18:59:58.755709 (+ 745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 18:59:58.759193 (+ 3484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 18:59:58.759200 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 18:59:58.759205 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 18:59:58.759568 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 18:59:58.759967 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 18:59:58.759971 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 18:59:58.759974 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 18:59:58.760124 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 18:59:58.760779 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 18:59:58.760785 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 18:59:58.760959 (+ 174us) client_negotiation.cc:769] Sending connection context
0924 18:59:58.761105 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 18:59:58.761459 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":50,"thread_start_us":424,"threads_started":1}
I20260924 18:59:58.763049 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 18:59:58 dist-test-slave-9fn1 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276398, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260924 18:59:58.795229 1718 heartbeater.cc:503] Master 127.0.97.62:41805 was elected leader, sending a full tablet report...
Sep 24 18:59:58 dist-test-slave-9fn1 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276398, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 18:59:58.806393 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:58.792911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39380 (local address 127.0.97.62:41805)
0924 18:59:58.793108 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:58.793114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:58.793388 (+ 274us) server_negotiation.cc:407] Connection header received
0924 18:59:58.793572 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:58.793578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:58.793651 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:58.793739 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:58.795093 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.795974 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.797438 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.797914 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.798010 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:58.801559 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:58.801566 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:58.801570 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:58.801609 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:58.803850 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:58.804475 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:58.804482 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:58.804486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:58.804602 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:58.804967 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:58.804974 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:58.804977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:58.805238 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:58.805484 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:58.805819 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 18:59:58.806050 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 18:59:58.810117 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 18:59:58.816856 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 18:59:58.843456 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:58.828467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48784 (local address 127.0.97.1:42503)
0924 18:59:58.828991 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:58.829001 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:58.829754 (+ 753us) server_negotiation.cc:407] Connection header received
0924 18:59:58.830019 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:58.830030 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:58.830342 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:58.830566 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 18:59:58.832932 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.836186 (+ 3254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.840877 (+ 4691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.842358 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.842498 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:58.842722 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:58.843028 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 18:59:58.843184 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":626,"threads_started":1}
I20260924 18:59:58.843464 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:58.828192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42503 (local address 127.0.0.1:48784)
0924 18:59:58.829580 (+ 1388us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:58.829614 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:58.829793 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:58.830894 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:58.830906 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:58.830946 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 18:59:58.832664 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:58.832684 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.836390 (+ 3706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.836398 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:58.840697 (+ 4299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:58.840708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.842558 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.842564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:58.842762 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:58.842809 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 18:59:58.842957 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 18:59:58.843141 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1141,"thread_start_us":12304,"threads_started":1}
I20260924 18:59:58.847777 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:58.828621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46661 (local address 127.0.0.1:44350)
0924 18:59:58.840588 (+ 11967us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:58.840607 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:58.840730 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:58.841468 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:58.841473 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:58.841515 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 18:59:58.841915 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:58.841927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.843579 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.843585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:58.845350 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:58.845368 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.847047 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.847053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:58.847208 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:58.847256 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 18:59:58.847346 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 18:59:58.847469 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11803,"thread_start_us":589,"threads_started":1}
I20260924 18:59:58.847839 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:58.828956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44350 (local address 127.0.97.2:46661)
0924 18:59:58.840444 (+ 11488us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:58.840456 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:58.840849 (+ 393us) server_negotiation.cc:407] Connection header received
0924 18:59:58.840969 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:58.840976 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:58.841206 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:58.841478 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 18:59:58.842101 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.843429 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.845618 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.846867 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.846990 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:58.847168 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:58.847417 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 18:59:58.847575 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10930,"thread_start_us":11006,"threads_started":1}
I20260924 18:59:58.847472 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 23b8240575d9442dbdf69667de6d520a (DEFAULT_TABLE table=test-table [id=daf4d60fe4a546af862b6fa276648ad5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 18:59:58.848841 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23b8240575d9442dbdf69667de6d520a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 18:59:58.849186 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:58.830018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42001 (local address 127.0.0.1:38608)
0924 18:59:58.841129 (+ 11111us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:58.841145 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:58.841286 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:58.841934 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:58.841939 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:58.841968 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 18:59:58.842338 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:58.842349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.845310 (+ 2961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.845313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:58.846633 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:58.846643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.848407 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.848412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:58.848576 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:58.848614 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 18:59:58.848707 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 18:59:58.848821 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10997,"mutex_wait_us":1,"thread_start_us":10882,"threads_started":1}
I20260924 18:59:58.849210 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:58.830337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38608 (local address 127.0.97.3:42001)
0924 18:59:58.839946 (+ 9609us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:58.839956 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:58.841248 (+ 1292us) server_negotiation.cc:407] Connection header received
0924 18:59:58.841433 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:58.841443 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:58.841726 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:58.841936 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 18:59:58.842567 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.845125 (+ 2558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.846785 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.848229 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:58.848369 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:58.848576 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:58.848759 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 18:59:58.848915 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9444,"thread_start_us":9275,"threads_started":1}
I20260924 18:59:58.851943 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 23b8240575d9442dbdf69667de6d520a (DEFAULT_TABLE table=test-table [id=daf4d60fe4a546af862b6fa276648ad5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 18:59:58.852994 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23b8240575d9442dbdf69667de6d520a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 18:59:58.853200 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 23b8240575d9442dbdf69667de6d520a (DEFAULT_TABLE table=test-table [id=daf4d60fe4a546af862b6fa276648ad5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 18:59:58.854545 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23b8240575d9442dbdf69667de6d520a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 18:59:58.863224 2015 tablet_bootstrap.cc:492] T 23b8240575d9442dbdf69667de6d520a P afb2353e1d634837966bb862c6e9753a: Bootstrap starting.
I20260924 18:59:58.866197 2016 tablet_bootstrap.cc:492] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a: Bootstrap starting.
I20260924 18:59:58.867017 2015 tablet_bootstrap.cc:654] T 23b8240575d9442dbdf69667de6d520a P afb2353e1d634837966bb862c6e9753a: Neither blocks nor log segments found. Creating new log.
I20260924 18:59:58.868661 2015 log.cc:824] T 23b8240575d9442dbdf69667de6d520a P afb2353e1d634837966bb862c6e9753a: Log is configured to *not* fsync() on all Append() calls
I20260924 18:59:58.870456 2016 tablet_bootstrap.cc:654] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a: Neither blocks nor log segments found. Creating new log.
I20260924 18:59:58.871333 2015 tablet_bootstrap.cc:492] T 23b8240575d9442dbdf69667de6d520a P afb2353e1d634837966bb862c6e9753a: No bootstrap required, opened a new log
I20260924 18:59:58.871387 2016 log.cc:824] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a: Log is configured to *not* fsync() on all Append() calls
I20260924 18:59:58.871611 2015 ts_tablet_manager.cc:1397] T 23b8240575d9442dbdf69667de6d520a P afb2353e1d634837966bb862c6e9753a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 18:59:58.873705 2016 tablet_bootstrap.cc:492] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a: No bootstrap required, opened a new log
I20260924 18:59:58.873687 2017 tablet_bootstrap.cc:492] T 23b8240575d9442dbdf69667de6d520a P bd36d282dbbe4a58a9895b149f77ace9: Bootstrap starting.
I20260924 18:59:58.873963 2016 ts_tablet_manager.cc:1397] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 18:59:58.877278 2017 tablet_bootstrap.cc:654] T 23b8240575d9442dbdf69667de6d520a P bd36d282dbbe4a58a9895b149f77ace9: Neither blocks nor log segments found. Creating new log.
I20260924 18:59:58.878371 2017 log.cc:824] T 23b8240575d9442dbdf69667de6d520a P bd36d282dbbe4a58a9895b149f77ace9: Log is configured to *not* fsync() on all Append() calls
I20260924 18:59:58.881142 2017 tablet_bootstrap.cc:492] T 23b8240575d9442dbdf69667de6d520a P bd36d282dbbe4a58a9895b149f77ace9: No bootstrap required, opened a new log
I20260924 18:59:58.881407 2017 ts_tablet_manager.cc:1397] T 23b8240575d9442dbdf69667de6d520a P bd36d282dbbe4a58a9895b149f77ace9: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.006s
I20260924 18:59:58.881249 2015 raft_consensus.cc:348] T 23b8240575d9442dbdf69667de6d520a P afb2353e1d634837966bb862c6e9753a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } }
I20260924 18:59:58.881727 2015 raft_consensus.cc:374] T 23b8240575d9442dbdf69667de6d520a P afb2353e1d634837966bb862c6e9753a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 18:59:58.881831 2015 raft_consensus.cc:729] T 23b8240575d9442dbdf69667de6d520a P afb2353e1d634837966bb862c6e9753a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afb2353e1d634837966bb862c6e9753a, State: Initialized, Role: FOLLOWER
I20260924 18:59:58.881994 2016 raft_consensus.cc:348] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } }
I20260924 18:59:58.882406 2016 raft_consensus.cc:374] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 18:59:58.882493 2016 raft_consensus.cc:729] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a402da4abc2a40309caa3ab7663c093a, State: Initialized, Role: FOLLOWER
I20260924 18:59:58.882416 2015 consensus_queue.cc:260] T 23b8240575d9442dbdf69667de6d520a P afb2353e1d634837966bb862c6e9753a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } }
I20260924 18:59:58.884111 2015 ts_tablet_manager.cc:1428] T 23b8240575d9442dbdf69667de6d520a P afb2353e1d634837966bb862c6e9753a: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260924 18:59:58.883157 2016 consensus_queue.cc:260] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } }
I20260924 18:59:58.885260 1857 heartbeater.cc:503] Master 127.0.97.62:41805 was elected leader, sending a full tablet report...
I20260924 18:59:58.885802 2016 ts_tablet_manager.cc:1428] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260924 18:59:58.891836 2017 raft_consensus.cc:348] T 23b8240575d9442dbdf69667de6d520a P bd36d282dbbe4a58a9895b149f77ace9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } }
I20260924 18:59:58.892122 2017 raft_consensus.cc:374] T 23b8240575d9442dbdf69667de6d520a P bd36d282dbbe4a58a9895b149f77ace9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 18:59:58.892186 2017 raft_consensus.cc:729] T 23b8240575d9442dbdf69667de6d520a P bd36d282dbbe4a58a9895b149f77ace9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd36d282dbbe4a58a9895b149f77ace9, State: Initialized, Role: FOLLOWER
I20260924 18:59:58.892661 2017 consensus_queue.cc:260] T 23b8240575d9442dbdf69667de6d520a P bd36d282dbbe4a58a9895b149f77ace9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } }
I20260924 18:59:58.894259 1996 heartbeater.cc:503] Master 127.0.97.62:41805 was elected leader, sending a full tablet report...
I20260924 18:59:58.894582 2017 ts_tablet_manager.cc:1428] T 23b8240575d9442dbdf69667de6d520a P bd36d282dbbe4a58a9895b149f77ace9: Time spent starting tablet: real 0.013s user 0.000s sys 0.013s
I20260924 18:59:58.994786 2022 raft_consensus.cc:482] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 18:59:58.995147 2022 raft_consensus.cc:504] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } }
I20260924 18:59:58.996965 2022 leader_election.cc:288] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers afb2353e1d634837966bb862c6e9753a (127.0.97.1:42503), bd36d282dbbe4a58a9895b149f77ace9 (127.0.97.3:42001)
I20260924 18:59:59.003578 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:58.997468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46831 (local address 127.0.97.3:42001)
0924 18:59:58.997672 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:58.997680 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:58.998230 (+ 550us) server_negotiation.cc:407] Connection header received
0924 18:59:58.998427 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:58.998432 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:58.998496 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:58.998577 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 18:59:58.999244 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.999973 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.001902 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.002608 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.002734 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:59.002866 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:59.003204 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 18:59:59.003318 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 18:59:59.003577 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:58.997318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46675 (local address 127.0.97.1:42503)
0924 18:59:58.997548 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:58.997557 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:58.998282 (+ 725us) server_negotiation.cc:407] Connection header received
0924 18:59:58.998479 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:58.998483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:58.998603 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:58.998697 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 18:59:58.999330 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:58.999996 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.001865 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.002617 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.002768 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:59.002893 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:59.003209 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 18:59:59.003323 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 18:59:59.003822 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:58.997267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42001 (local address 127.0.97.2:46831)
0924 18:59:58.998119 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:58.998137 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:58.998280 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:58.998626 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:58.998632 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:58.998661 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 18:59:58.999098 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:58.999106 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.000102 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.000107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:59.001741 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:59.001753 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.002827 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.002834 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:59.003028 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:59.003057 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 18:59:59.003183 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 18:59:59.003307 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":179,"thread_start_us":488,"threads_started":1}
I20260924 18:59:59.003968 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:58.997308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42503 (local address 127.0.97.2:46675)
0924 18:59:58.998140 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:58.998154 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:58.998306 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:58.998742 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:58.998747 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:58.998781 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 18:59:58.999154 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:58.999182 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.000145 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.000149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:59.001718 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:59.001732 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.002871 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.002877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:59.003063 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:59.003083 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 18:59:59.003182 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 18:59:59.003318 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":90,"thread_start_us":530,"threads_started":1}
W20260924 18:59:59.005331 1997 tablet.cc:2369] T 23b8240575d9442dbdf69667de6d520a P bd36d282dbbe4a58a9895b149f77ace9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 18:59:59.005260 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b8240575d9442dbdf69667de6d520a" candidate_uuid: "a402da4abc2a40309caa3ab7663c093a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afb2353e1d634837966bb862c6e9753a" is_pre_election: true
I20260924 18:59:59.005247 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b8240575d9442dbdf69667de6d520a" candidate_uuid: "a402da4abc2a40309caa3ab7663c093a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd36d282dbbe4a58a9895b149f77ace9" is_pre_election: true
I20260924 18:59:59.005769 1673 raft_consensus.cc:2454] T 23b8240575d9442dbdf69667de6d520a P afb2353e1d634837966bb862c6e9753a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a402da4abc2a40309caa3ab7663c093a in term 0.
I20260924 18:59:59.005854 1946 raft_consensus.cc:2454] T 23b8240575d9442dbdf69667de6d520a P bd36d282dbbe4a58a9895b149f77ace9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a402da4abc2a40309caa3ab7663c093a in term 0.
I20260924 18:59:59.006436 1740 leader_election.cc:302] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a402da4abc2a40309caa3ab7663c093a, afb2353e1d634837966bb862c6e9753a, bd36d282dbbe4a58a9895b149f77ace9; no voters:
I20260924 18:59:59.006798 2022 raft_consensus.cc:2781] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 18:59:59.006947 2022 raft_consensus.cc:482] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 18:59:59.007047 2022 raft_consensus.cc:3037] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a [term 0 FOLLOWER]: Advancing to term 1
I20260924 18:59:59.009444 2022 raft_consensus.cc:504] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } }
I20260924 18:59:59.010150 2022 leader_election.cc:288] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a [CANDIDATE]: Term 1 election: Requested vote from peers afb2353e1d634837966bb862c6e9753a (127.0.97.1:42503), bd36d282dbbe4a58a9895b149f77ace9 (127.0.97.3:42001)
I20260924 18:59:59.010490 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b8240575d9442dbdf69667de6d520a" candidate_uuid: "a402da4abc2a40309caa3ab7663c093a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afb2353e1d634837966bb862c6e9753a"
I20260924 18:59:59.010550 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b8240575d9442dbdf69667de6d520a" candidate_uuid: "a402da4abc2a40309caa3ab7663c093a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd36d282dbbe4a58a9895b149f77ace9"
I20260924 18:59:59.010699 1673 raft_consensus.cc:3037] T 23b8240575d9442dbdf69667de6d520a P afb2353e1d634837966bb862c6e9753a [term 0 FOLLOWER]: Advancing to term 1
I20260924 18:59:59.010732 1946 raft_consensus.cc:3037] T 23b8240575d9442dbdf69667de6d520a P bd36d282dbbe4a58a9895b149f77ace9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 18:59:59.013854 1946 raft_consensus.cc:2454] T 23b8240575d9442dbdf69667de6d520a P bd36d282dbbe4a58a9895b149f77ace9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a402da4abc2a40309caa3ab7663c093a in term 1.
I20260924 18:59:59.013851 1673 raft_consensus.cc:2454] T 23b8240575d9442dbdf69667de6d520a P afb2353e1d634837966bb862c6e9753a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a402da4abc2a40309caa3ab7663c093a in term 1.
I20260924 18:59:59.014232 1740 leader_election.cc:302] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a402da4abc2a40309caa3ab7663c093a, afb2353e1d634837966bb862c6e9753a; no voters:
I20260924 18:59:59.014530 2022 raft_consensus.cc:2781] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 18:59:59.014951 2022 raft_consensus.cc:686] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a [term 1 LEADER]: Becoming Leader. State: Replica: a402da4abc2a40309caa3ab7663c093a, State: Running, Role: LEADER
I20260924 18:59:59.015476 2022 consensus_queue.cc:237] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } }
I20260924 18:59:59.019809 1510 catalog_manager.cc:5495] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a reported cstate change: term changed from 0 to 1, leader changed from <none> to a402da4abc2a40309caa3ab7663c093a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "a402da4abc2a40309caa3ab7663c093a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } health_report { overall_health: UNKNOWN } } }
W20260924 18:59:59.028323 1858 tablet.cc:2369] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 18:59:59.038071 1719 tablet.cc:2369] T 23b8240575d9442dbdf69667de6d520a P afb2353e1d634837966bb862c6e9753a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 18:59:59.078403 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:59.073738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44360 (local address 127.0.97.2:46661)
0924 18:59:59.073920 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:59.073924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:59.073941 (+ 17us) server_negotiation.cc:407] Connection header received
0924 18:59:59.074168 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:59.074171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:59.074254 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:59.074370 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 18:59:59.074641 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.075261 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.076445 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.076827 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.076909 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:59.077318 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 18:59:59.077841 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 18:59:59.077939 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:59.078048 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 18:59:59.078105 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 18:59:59.085829 1946 raft_consensus.cc:1261] T 23b8240575d9442dbdf69667de6d520a P bd36d282dbbe4a58a9895b149f77ace9 [term 1 FOLLOWER]: Refusing update from remote peer a402da4abc2a40309caa3ab7663c093a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 18:59:59.085816 1673 raft_consensus.cc:1261] T 23b8240575d9442dbdf69667de6d520a P afb2353e1d634837966bb862c6e9753a [term 1 FOLLOWER]: Refusing update from remote peer a402da4abc2a40309caa3ab7663c093a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 18:59:59.086732 2026 consensus_queue.cc:1035] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 18:59:59.087113 2022 consensus_queue.cc:1035] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a [LEADER]: Connected to new peer: Peer: permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 18:59:59.121078 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 18:59:59.121382 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 18:59:59.125023 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24931820b37e4f298ea207374672357e: Sending DeleteTablet for 3 replicas of tablet 23b8240575d9442dbdf69667de6d520a
I20260924 18:59:59.126173 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 23b8240575d9442dbdf69667de6d520a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 18:59:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48784
I20260924 18:59:59.126296 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet 23b8240575d9442dbdf69667de6d520a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 18:59:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38608
I20260924 18:59:59.126899 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 23b8240575d9442dbdf69667de6d520a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 18:59:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44350
I20260924 18:59:59.127101 2039 tablet_replica.cc:330] T 23b8240575d9442dbdf69667de6d520a P afb2353e1d634837966bb862c6e9753a: stopping tablet replica
I20260924 18:59:59.127556 2039 raft_consensus.cc:2229] T 23b8240575d9442dbdf69667de6d520a P afb2353e1d634837966bb862c6e9753a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 18:59:59.128069 2039 raft_consensus.cc:2258] T 23b8240575d9442dbdf69667de6d520a P afb2353e1d634837966bb862c6e9753a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 18:59:59.129029 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 18:59:59.130081 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 18:59:59.133672 2039 ts_tablet_manager.cc:1905] T 23b8240575d9442dbdf69667de6d520a P afb2353e1d634837966bb862c6e9753a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 18:59:59.138782 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 78fa364e04e34a54ad9393d7efa9fed2 (DEFAULT_TABLE table=test-table [id=4b75e9e1d12c42a0b928e73190c1f614]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 18:59:59.139231 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78fa364e04e34a54ad9393d7efa9fed2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 18:59:59.139477 2041 tablet_replica.cc:330] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a: stopping tablet replica
I20260924 18:59:59.139420 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 78fa364e04e34a54ad9393d7efa9fed2 (DEFAULT_TABLE table=test-table [id=4b75e9e1d12c42a0b928e73190c1f614]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 18:59:59.139956 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78fa364e04e34a54ad9393d7efa9fed2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 18:59:59.139986 2041 raft_consensus.cc:2229] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a [term 1 LEADER]: Raft consensus shutting down.
I20260924 18:59:59.140745 2041 raft_consensus.cc:2258] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 18:59:59.141532 2040 tablet_replica.cc:330] T 23b8240575d9442dbdf69667de6d520a P bd36d282dbbe4a58a9895b149f77ace9: stopping tablet replica
I20260924 18:59:59.141985 2040 raft_consensus.cc:2229] T 23b8240575d9442dbdf69667de6d520a P bd36d282dbbe4a58a9895b149f77ace9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 18:59:59.142016 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 78fa364e04e34a54ad9393d7efa9fed2 (DEFAULT_TABLE table=test-table [id=4b75e9e1d12c42a0b928e73190c1f614]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 18:59:59.142529 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78fa364e04e34a54ad9393d7efa9fed2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 18:59:59.142561 2040 raft_consensus.cc:2258] T 23b8240575d9442dbdf69667de6d520a P bd36d282dbbe4a58a9895b149f77ace9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 18:59:59.144618 2040 ts_tablet_manager.cc:1905] T 23b8240575d9442dbdf69667de6d520a P bd36d282dbbe4a58a9895b149f77ace9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 18:59:59.144860 2041 ts_tablet_manager.cc:1905] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 18:59:59.153162 2017 tablet_bootstrap.cc:492] T 78fa364e04e34a54ad9393d7efa9fed2 P bd36d282dbbe4a58a9895b149f77ace9: Bootstrap starting.
I20260924 18:59:59.155143 2040 ts_tablet_manager.cc:1918] T 23b8240575d9442dbdf69667de6d520a P bd36d282dbbe4a58a9895b149f77ace9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 18:59:59.155241 2017 tablet_bootstrap.cc:654] T 78fa364e04e34a54ad9393d7efa9fed2 P bd36d282dbbe4a58a9895b149f77ace9: Neither blocks nor log segments found. Creating new log.
I20260924 18:59:59.155404 2040 log.cc:1196] T 23b8240575d9442dbdf69667de6d520a P bd36d282dbbe4a58a9895b149f77ace9: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-2/wal/wals/23b8240575d9442dbdf69667de6d520a
I20260924 18:59:59.155967 2040 ts_tablet_manager.cc:1939] T 23b8240575d9442dbdf69667de6d520a P bd36d282dbbe4a58a9895b149f77ace9: Deleting consensus metadata
I20260924 18:59:59.158593 1496 catalog_manager.cc:4841] TS bd36d282dbbe4a58a9895b149f77ace9 (127.0.97.3:42001): tablet 23b8240575d9442dbdf69667de6d520a (table test-table [id=daf4d60fe4a546af862b6fa276648ad5]) successfully deleted
I20260924 18:59:59.163566 2041 ts_tablet_manager.cc:1918] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 18:59:59.163635 2015 tablet_bootstrap.cc:492] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a: Bootstrap starting.
I20260924 18:59:59.163735 2041 log.cc:1196] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-1/wal/wals/23b8240575d9442dbdf69667de6d520a
I20260924 18:59:59.163787 2016 tablet_bootstrap.cc:492] T 78fa364e04e34a54ad9393d7efa9fed2 P a402da4abc2a40309caa3ab7663c093a: Bootstrap starting.
I20260924 18:59:59.164266 2041 ts_tablet_manager.cc:1939] T 23b8240575d9442dbdf69667de6d520a P a402da4abc2a40309caa3ab7663c093a: Deleting consensus metadata
I20260924 18:59:59.165747 2016 tablet_bootstrap.cc:654] T 78fa364e04e34a54ad9393d7efa9fed2 P a402da4abc2a40309caa3ab7663c093a: Neither blocks nor log segments found. Creating new log.
I20260924 18:59:59.166082 2015 tablet_bootstrap.cc:654] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a: Neither blocks nor log segments found. Creating new log.
I20260924 18:59:59.169216 2017 tablet_bootstrap.cc:492] T 78fa364e04e34a54ad9393d7efa9fed2 P bd36d282dbbe4a58a9895b149f77ace9: No bootstrap required, opened a new log
I20260924 18:59:59.169379 2017 ts_tablet_manager.cc:1397] T 78fa364e04e34a54ad9393d7efa9fed2 P bd36d282dbbe4a58a9895b149f77ace9: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.004s
I20260924 18:59:59.170511 2017 raft_consensus.cc:348] T 78fa364e04e34a54ad9393d7efa9fed2 P bd36d282dbbe4a58a9895b149f77ace9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } }
I20260924 18:59:59.170732 2017 raft_consensus.cc:374] T 78fa364e04e34a54ad9393d7efa9fed2 P bd36d282dbbe4a58a9895b149f77ace9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 18:59:59.170821 2017 raft_consensus.cc:729] T 78fa364e04e34a54ad9393d7efa9fed2 P bd36d282dbbe4a58a9895b149f77ace9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd36d282dbbe4a58a9895b149f77ace9, State: Initialized, Role: FOLLOWER
I20260924 18:59:59.171110 2017 consensus_queue.cc:260] T 78fa364e04e34a54ad9393d7efa9fed2 P bd36d282dbbe4a58a9895b149f77ace9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } }
I20260924 18:59:59.174328 2017 ts_tablet_manager.cc:1428] T 78fa364e04e34a54ad9393d7efa9fed2 P bd36d282dbbe4a58a9895b149f77ace9: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260924 18:59:59.177706 1496 catalog_manager.cc:4841] TS a402da4abc2a40309caa3ab7663c093a (127.0.97.2:46661): tablet 23b8240575d9442dbdf69667de6d520a (table test-table [id=daf4d60fe4a546af862b6fa276648ad5]) successfully deleted
I20260924 18:59:59.178612 2016 tablet_bootstrap.cc:492] T 78fa364e04e34a54ad9393d7efa9fed2 P a402da4abc2a40309caa3ab7663c093a: No bootstrap required, opened a new log
I20260924 18:59:59.178759 2016 ts_tablet_manager.cc:1397] T 78fa364e04e34a54ad9393d7efa9fed2 P a402da4abc2a40309caa3ab7663c093a: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.011s
I20260924 18:59:59.179540 2016 raft_consensus.cc:348] T 78fa364e04e34a54ad9393d7efa9fed2 P a402da4abc2a40309caa3ab7663c093a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } }
I20260924 18:59:59.179754 2016 raft_consensus.cc:374] T 78fa364e04e34a54ad9393d7efa9fed2 P a402da4abc2a40309caa3ab7663c093a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 18:59:59.179844 2016 raft_consensus.cc:729] T 78fa364e04e34a54ad9393d7efa9fed2 P a402da4abc2a40309caa3ab7663c093a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a402da4abc2a40309caa3ab7663c093a, State: Initialized, Role: FOLLOWER
I20260924 18:59:59.180102 2016 consensus_queue.cc:260] T 78fa364e04e34a54ad9393d7efa9fed2 P a402da4abc2a40309caa3ab7663c093a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } }
I20260924 18:59:59.180845 2016 ts_tablet_manager.cc:1428] T 78fa364e04e34a54ad9393d7efa9fed2 P a402da4abc2a40309caa3ab7663c093a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 18:59:59.181579 2039 ts_tablet_manager.cc:1918] T 23b8240575d9442dbdf69667de6d520a P afb2353e1d634837966bb862c6e9753a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 18:59:59.181773 2039 log.cc:1196] T 23b8240575d9442dbdf69667de6d520a P afb2353e1d634837966bb862c6e9753a: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-0/wal/wals/23b8240575d9442dbdf69667de6d520a
I20260924 18:59:59.182294 2039 ts_tablet_manager.cc:1939] T 23b8240575d9442dbdf69667de6d520a P afb2353e1d634837966bb862c6e9753a: Deleting consensus metadata
I20260924 18:59:59.184044 1497 catalog_manager.cc:4841] TS afb2353e1d634837966bb862c6e9753a (127.0.97.1:42503): tablet 23b8240575d9442dbdf69667de6d520a (table test-table [id=daf4d60fe4a546af862b6fa276648ad5]) successfully deleted
I20260924 18:59:59.185293 2015 tablet_bootstrap.cc:492] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a: No bootstrap required, opened a new log
I20260924 18:59:59.185439 2015 ts_tablet_manager.cc:1397] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.009s
I20260924 18:59:59.186219 2015 raft_consensus.cc:348] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } }
I20260924 18:59:59.186439 2015 raft_consensus.cc:374] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 18:59:59.186532 2015 raft_consensus.cc:729] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afb2353e1d634837966bb862c6e9753a, State: Initialized, Role: FOLLOWER
I20260924 18:59:59.186848 2015 consensus_queue.cc:260] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } }
I20260924 18:59:59.187413 2015 ts_tablet_manager.cc:1428] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 18:59:59.422875 2021 raft_consensus.cc:482] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 18:59:59.423118 2021 raft_consensus.cc:504] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } }
I20260924 18:59:59.424779 2021 leader_election.cc:288] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a402da4abc2a40309caa3ab7663c093a (127.0.97.2:46661), bd36d282dbbe4a58a9895b149f77ace9 (127.0.97.3:42001)
I20260924 18:59:59.431984 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:59.425895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47721 (local address 127.0.97.3:42001)
0924 18:59:59.426084 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:59.426090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:59.426700 (+ 610us) server_negotiation.cc:407] Connection header received
0924 18:59:59.426888 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:59.426893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:59.426974 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:59.427081 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 18:59:59.427549 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.428229 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.430175 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.431026 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.431150 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:59.431255 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:59.431595 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 18:59:59.431722 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 18:59:59.432469 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:59.425719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42001 (local address 127.0.97.1:47721)
0924 18:59:59.426564 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:59.426586 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:59.426735 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:59.427092 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:59.427096 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:59.427117 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 18:59:59.427413 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:59.427420 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.428351 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.428359 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:59.430024 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:59.430035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.431235 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.431242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:59.431419 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:59.431454 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 18:59:59.431559 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 18:59:59.432070 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":98,"thread_start_us":512,"threads_started":1}
I20260924 18:59:59.433243 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78fa364e04e34a54ad9393d7efa9fed2" candidate_uuid: "afb2353e1d634837966bb862c6e9753a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd36d282dbbe4a58a9895b149f77ace9" is_pre_election: true
I20260924 18:59:59.433527 1946 raft_consensus.cc:2454] T 78fa364e04e34a54ad9393d7efa9fed2 P bd36d282dbbe4a58a9895b149f77ace9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afb2353e1d634837966bb862c6e9753a in term 0.
I20260924 18:59:59.433578 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:59.424998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46661 (local address 127.0.97.1:54339)
0924 18:59:59.425597 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:59.425617 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:59.425831 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:59.426446 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:59.426449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:59.426470 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 18:59:59.426834 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:59.426846 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.428421 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.428428 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:59.430705 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:59.430721 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.432527 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.432829 (+ 302us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:59.432955 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:59.432977 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 18:59:59.433077 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 18:59:59.433200 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":32,"thread_start_us":289,"threads_started":1}
I20260924 18:59:59.434016 1599 leader_election.cc:302] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afb2353e1d634837966bb862c6e9753a, bd36d282dbbe4a58a9895b149f77ace9; no voters:
I20260924 18:59:59.434293 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:59.425101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54339 (local address 127.0.97.2:46661)
0924 18:59:59.425302 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:59.425310 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:59.425709 (+ 399us) server_negotiation.cc:407] Connection header received
0924 18:59:59.426045 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:59.426054 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:59.426151 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:59.426227 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 18:59:59.427028 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.428224 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.430900 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.432316 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.432444 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:59.432540 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:59.433893 (+ 1353us) server_negotiation.cc:299] Negotiation successful
0924 18:59:59.434034 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 18:59:59.434422 2021 raft_consensus.cc:2781] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 18:59:59.434551 2021 raft_consensus.cc:482] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 18:59:59.434633 2021 raft_consensus.cc:3037] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a [term 0 FOLLOWER]: Advancing to term 1
I20260924 18:59:59.435055 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78fa364e04e34a54ad9393d7efa9fed2" candidate_uuid: "afb2353e1d634837966bb862c6e9753a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a402da4abc2a40309caa3ab7663c093a" is_pre_election: true
I20260924 18:59:59.435438 1812 raft_consensus.cc:2454] T 78fa364e04e34a54ad9393d7efa9fed2 P a402da4abc2a40309caa3ab7663c093a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afb2353e1d634837966bb862c6e9753a in term 0.
I20260924 18:59:59.437850 2021 raft_consensus.cc:504] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } }
I20260924 18:59:59.438575 2021 leader_election.cc:288] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a [CANDIDATE]: Term 1 election: Requested vote from peers a402da4abc2a40309caa3ab7663c093a (127.0.97.2:46661), bd36d282dbbe4a58a9895b149f77ace9 (127.0.97.3:42001)
I20260924 18:59:59.439474 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78fa364e04e34a54ad9393d7efa9fed2" candidate_uuid: "afb2353e1d634837966bb862c6e9753a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a402da4abc2a40309caa3ab7663c093a"
I20260924 18:59:59.439502 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78fa364e04e34a54ad9393d7efa9fed2" candidate_uuid: "afb2353e1d634837966bb862c6e9753a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd36d282dbbe4a58a9895b149f77ace9"
I20260924 18:59:59.439683 1946 raft_consensus.cc:3037] T 78fa364e04e34a54ad9393d7efa9fed2 P bd36d282dbbe4a58a9895b149f77ace9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 18:59:59.439704 1812 raft_consensus.cc:3037] T 78fa364e04e34a54ad9393d7efa9fed2 P a402da4abc2a40309caa3ab7663c093a [term 0 FOLLOWER]: Advancing to term 1
I20260924 18:59:59.442549 1946 raft_consensus.cc:2454] T 78fa364e04e34a54ad9393d7efa9fed2 P bd36d282dbbe4a58a9895b149f77ace9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afb2353e1d634837966bb862c6e9753a in term 1.
I20260924 18:59:59.443377 1812 raft_consensus.cc:2454] T 78fa364e04e34a54ad9393d7efa9fed2 P a402da4abc2a40309caa3ab7663c093a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afb2353e1d634837966bb862c6e9753a in term 1.
I20260924 18:59:59.443678 1599 leader_election.cc:302] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afb2353e1d634837966bb862c6e9753a, bd36d282dbbe4a58a9895b149f77ace9; no voters:
I20260924 18:59:59.443974 2021 raft_consensus.cc:2781] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 18:59:59.444418 2021 raft_consensus.cc:686] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a [term 1 LEADER]: Becoming Leader. State: Replica: afb2353e1d634837966bb862c6e9753a, State: Running, Role: LEADER
I20260924 18:59:59.444828 2021 consensus_queue.cc:237] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } }
I20260924 18:59:59.448305 1511 catalog_manager.cc:5495] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a reported cstate change: term changed from 0 to 1, leader changed from <none> to afb2353e1d634837966bb862c6e9753a (127.0.97.1). New cstate: current_term: 1 leader_uuid: "afb2353e1d634837966bb862c6e9753a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } health_report { overall_health: UNKNOWN } } }
Sep 24 18:59:59 dist-test-slave-9fn1 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276397, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 18:59:59.554842 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:59.539951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41805 (local address 127.0.0.1:39394)
0924 18:59:59.540651 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:59.540672 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:59.540817 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:59.541224 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:59.541229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:59.542050 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0924 18:59:59.542284 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:59.542295 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.543362 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.543366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:59.545214 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:59.545228 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.545943 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.545953 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:59.546132 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:59.546844 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 18:59:59.546865 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 18:59:59.549458 (+ 2593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 18:59:59.552423 (+ 2965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 18:59:59.552431 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 18:59:59.552437 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 18:59:59.552830 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 18:59:59.553197 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 18:59:59.553199 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 18:59:59.553202 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 18:59:59.553378 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 18:59:59.554032 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 18:59:59.554038 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 18:59:59.554166 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 18:59:59.554306 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 18:59:59.554512 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":369,"threads_started":1}
I20260924 18:59:59.554989 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:59.539950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39394 (local address 127.0.97.62:41805)
0924 18:59:59.540950 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:59.540954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:59.540973 (+ 19us) server_negotiation.cc:407] Connection header received
0924 18:59:59.541017 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:59.541020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:59.541069 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:59.541164 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:59.542450 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.543215 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.545346 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.545793 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.545886 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:59.549687 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:59.549692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:59.549695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:59.549721 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:59.552322 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:59.552960 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:59.552966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:59.552972 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:59.553103 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:59.553495 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:59.553642 (+ 147us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:59.553650 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:59.553911 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:59.554017 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:59.554400 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 18:59:59.554628 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":911,"spinlock_wait_cycles":22272,"thread_start_us":779,"threads_started":1}
I20260924 18:59:59.559644 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39394:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 18:59:59.560547 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 18:59:59.566512 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 7795765281244264856c4434a3af6fab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f9c6515fccf94e5a872f22a32c8a58e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 18:59:59.566660 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 7795765281244264856c4434a3af6fab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f9c6515fccf94e5a872f22a32c8a58e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 18:59:59.566811 1925 tablet_service.cc:1467] Processing CreateTablet for tablet 7795765281244264856c4434a3af6fab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f9c6515fccf94e5a872f22a32c8a58e2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 18:59:59.567077 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7795765281244264856c4434a3af6fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 18:59:59.567216 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7795765281244264856c4434a3af6fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 18:59:59.567286 1925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7795765281244264856c4434a3af6fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 18:59:59.572413 2016 tablet_bootstrap.cc:492] T 7795765281244264856c4434a3af6fab P a402da4abc2a40309caa3ab7663c093a: Bootstrap starting.
I20260924 18:59:59.573854 2016 tablet_bootstrap.cc:654] T 7795765281244264856c4434a3af6fab P a402da4abc2a40309caa3ab7663c093a: Neither blocks nor log segments found. Creating new log.
I20260924 18:59:59.574399 2017 tablet_bootstrap.cc:492] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9: Bootstrap starting.
I20260924 18:59:59.574450 2015 tablet_bootstrap.cc:492] T 7795765281244264856c4434a3af6fab P afb2353e1d634837966bb862c6e9753a: Bootstrap starting.
I20260924 18:59:59.575825 2017 tablet_bootstrap.cc:654] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9: Neither blocks nor log segments found. Creating new log.
I20260924 18:59:59.575896 2016 tablet_bootstrap.cc:492] T 7795765281244264856c4434a3af6fab P a402da4abc2a40309caa3ab7663c093a: No bootstrap required, opened a new log
I20260924 18:59:59.575973 2015 tablet_bootstrap.cc:654] T 7795765281244264856c4434a3af6fab P afb2353e1d634837966bb862c6e9753a: Neither blocks nor log segments found. Creating new log.
I20260924 18:59:59.576036 2016 ts_tablet_manager.cc:1397] T 7795765281244264856c4434a3af6fab P a402da4abc2a40309caa3ab7663c093a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 18:59:59.576719 2016 raft_consensus.cc:348] T 7795765281244264856c4434a3af6fab P a402da4abc2a40309caa3ab7663c093a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } }
I20260924 18:59:59.576964 2016 raft_consensus.cc:374] T 7795765281244264856c4434a3af6fab P a402da4abc2a40309caa3ab7663c093a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 18:59:59.577040 2016 raft_consensus.cc:729] T 7795765281244264856c4434a3af6fab P a402da4abc2a40309caa3ab7663c093a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a402da4abc2a40309caa3ab7663c093a, State: Initialized, Role: FOLLOWER
I20260924 18:59:59.577358 2016 consensus_queue.cc:260] T 7795765281244264856c4434a3af6fab P a402da4abc2a40309caa3ab7663c093a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } }
I20260924 18:59:59.577975 2017 tablet_bootstrap.cc:492] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9: No bootstrap required, opened a new log
I20260924 18:59:59.578122 2017 ts_tablet_manager.cc:1397] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260924 18:59:59.577951 2022 tablet_replica.cc:439] T 7795765281244264856c4434a3af6fab P a402da4abc2a40309caa3ab7663c093a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } } }
I20260924 18:59:59.578261 2022 tablet_replica.cc:442] T 7795765281244264856c4434a3af6fab P a402da4abc2a40309caa3ab7663c093a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 18:59:59.578823 2017 raft_consensus.cc:348] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } }
I20260924 18:59:59.578939 2016 ts_tablet_manager.cc:1428] T 7795765281244264856c4434a3af6fab P a402da4abc2a40309caa3ab7663c093a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 18:59:59.579033 2017 raft_consensus.cc:374] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 18:59:59.579104 2017 raft_consensus.cc:729] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd36d282dbbe4a58a9895b149f77ace9, State: Initialized, Role: FOLLOWER
I20260924 18:59:59.579361 2017 consensus_queue.cc:260] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } }
I20260924 18:59:59.579883 2017 ts_tablet_manager.cc:1428] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 18:59:59.579916 2023 tablet_replica.cc:439] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } } }
I20260924 18:59:59.580268 2023 tablet_replica.cc:442] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 18:59:59.581260 2015 tablet_bootstrap.cc:492] T 7795765281244264856c4434a3af6fab P afb2353e1d634837966bb862c6e9753a: No bootstrap required, opened a new log
I20260924 18:59:59.581393 2015 ts_tablet_manager.cc:1397] T 7795765281244264856c4434a3af6fab P afb2353e1d634837966bb862c6e9753a: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260924 18:59:59.582069 2015 raft_consensus.cc:348] T 7795765281244264856c4434a3af6fab P afb2353e1d634837966bb862c6e9753a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } }
I20260924 18:59:59.582293 2015 raft_consensus.cc:374] T 7795765281244264856c4434a3af6fab P afb2353e1d634837966bb862c6e9753a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 18:59:59.582391 2015 raft_consensus.cc:729] T 7795765281244264856c4434a3af6fab P afb2353e1d634837966bb862c6e9753a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afb2353e1d634837966bb862c6e9753a, State: Initialized, Role: FOLLOWER
I20260924 18:59:59.582659 2015 consensus_queue.cc:260] T 7795765281244264856c4434a3af6fab P afb2353e1d634837966bb862c6e9753a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } }
I20260924 18:59:59.583199 2015 ts_tablet_manager.cc:1428] T 7795765281244264856c4434a3af6fab P afb2353e1d634837966bb862c6e9753a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 18:59:59.583072 2021 tablet_replica.cc:439] T 7795765281244264856c4434a3af6fab P afb2353e1d634837966bb862c6e9753a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } } }
I20260924 18:59:59.583341 2021 tablet_replica.cc:442] T 7795765281244264856c4434a3af6fab P afb2353e1d634837966bb862c6e9753a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 18:59:59.660884 2023 raft_consensus.cc:482] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 18:59:59.661096 2023 raft_consensus.cc:504] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } }
I20260924 18:59:59.662608 2023 leader_election.cc:288] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers afb2353e1d634837966bb862c6e9753a (127.0.97.1:42503), a402da4abc2a40309caa3ab7663c093a (127.0.97.2:46661)
I20260924 18:59:59.669409 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:59.662892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:32865 (local address 127.0.97.1:42503)
0924 18:59:59.663962 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:59.663967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:59.663981 (+ 14us) server_negotiation.cc:407] Connection header received
0924 18:59:59.664034 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:59.664037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:59.664088 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:59.664194 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 18:59:59.664610 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.665352 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.667556 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.668332 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.668473 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:59.668577 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:59.668940 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 18:59:59.669092 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":948,"thread_start_us":837,"threads_started":1}
I20260924 18:59:59.669600 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:59.663033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50829 (local address 127.0.97.2:46661)
0924 18:59:59.663262 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:59.663271 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:59.664272 (+ 1001us) server_negotiation.cc:407] Connection header received
0924 18:59:59.664471 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:59.664481 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:59.664531 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:59.664622 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 18:59:59.665080 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.665912 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.667956 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.668677 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.668794 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:59.668906 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:59.669216 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 18:59:59.669346 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 18:59:59.669430 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:59.662832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42503 (local address 127.0.97.3:32865)
0924 18:59:59.663648 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:59.663668 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:59.663822 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:59.664211 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:59.664216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:59.664242 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 18:59:59.664470 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:59.664478 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.665493 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.665527 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:59.667408 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:59.667419 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.668504 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.668510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:59.668668 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:59.668691 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 18:59:59.668798 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 18:59:59.669128 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":603,"threads_started":1}
I20260924 18:59:59.670246 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:59.662889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46661 (local address 127.0.97.3:50829)
0924 18:59:59.664168 (+ 1279us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:59.664182 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:59.664340 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:59.664633 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:59.664637 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:59.664657 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 18:59:59.664935 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:59.664943 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.666031 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.666037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:59.667803 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:59.667814 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.668889 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.668894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:59.669050 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:59.669080 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 18:59:59.669823 (+ 743us) client_negotiation.cc:240] Negotiation successful
0924 18:59:59.669961 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1178,"mutex_wait_us":1,"thread_start_us":1043,"threads_started":1}
I20260924 18:59:59.670488 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7795765281244264856c4434a3af6fab" candidate_uuid: "bd36d282dbbe4a58a9895b149f77ace9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afb2353e1d634837966bb862c6e9753a" is_pre_election: true
I20260924 18:59:59.670766 1673 raft_consensus.cc:2454] T 7795765281244264856c4434a3af6fab P afb2353e1d634837966bb862c6e9753a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd36d282dbbe4a58a9895b149f77ace9 in term 0.
I20260924 18:59:59.670925 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7795765281244264856c4434a3af6fab" candidate_uuid: "bd36d282dbbe4a58a9895b149f77ace9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a402da4abc2a40309caa3ab7663c093a" is_pre_election: true
I20260924 18:59:59.671161 1812 raft_consensus.cc:2454] T 7795765281244264856c4434a3af6fab P a402da4abc2a40309caa3ab7663c093a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd36d282dbbe4a58a9895b149f77ace9 in term 0.
I20260924 18:59:59.671234 1879 leader_election.cc:302] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afb2353e1d634837966bb862c6e9753a, bd36d282dbbe4a58a9895b149f77ace9; no voters:
I20260924 18:59:59.671640 2023 raft_consensus.cc:2781] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 18:59:59.671746 2023 raft_consensus.cc:482] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 18:59:59.671859 2023 raft_consensus.cc:3037] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 18:59:59.673969 2023 raft_consensus.cc:504] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } }
I20260924 18:59:59.674515 2023 leader_election.cc:288] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9 [CANDIDATE]: Term 1 election: Requested vote from peers afb2353e1d634837966bb862c6e9753a (127.0.97.1:42503), a402da4abc2a40309caa3ab7663c093a (127.0.97.2:46661)
I20260924 18:59:59.674921 1812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7795765281244264856c4434a3af6fab" candidate_uuid: "bd36d282dbbe4a58a9895b149f77ace9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a402da4abc2a40309caa3ab7663c093a"
I20260924 18:59:59.674971 1673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7795765281244264856c4434a3af6fab" candidate_uuid: "bd36d282dbbe4a58a9895b149f77ace9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afb2353e1d634837966bb862c6e9753a"
I20260924 18:59:59.675130 1812 raft_consensus.cc:3037] T 7795765281244264856c4434a3af6fab P a402da4abc2a40309caa3ab7663c093a [term 0 FOLLOWER]: Advancing to term 1
I20260924 18:59:59.675158 1673 raft_consensus.cc:3037] T 7795765281244264856c4434a3af6fab P afb2353e1d634837966bb862c6e9753a [term 0 FOLLOWER]: Advancing to term 1
I20260924 18:59:59.677067 1673 raft_consensus.cc:2454] T 7795765281244264856c4434a3af6fab P afb2353e1d634837966bb862c6e9753a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd36d282dbbe4a58a9895b149f77ace9 in term 1.
I20260924 18:59:59.677048 1812 raft_consensus.cc:2454] T 7795765281244264856c4434a3af6fab P a402da4abc2a40309caa3ab7663c093a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd36d282dbbe4a58a9895b149f77ace9 in term 1.
I20260924 18:59:59.677450 1879 leader_election.cc:302] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: afb2353e1d634837966bb862c6e9753a, bd36d282dbbe4a58a9895b149f77ace9; no voters:
I20260924 18:59:59.677778 2023 raft_consensus.cc:2781] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 18:59:59.678167 2023 raft_consensus.cc:686] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9 [term 1 LEADER]: Becoming Leader. State: Replica: bd36d282dbbe4a58a9895b149f77ace9, State: Running, Role: LEADER
I20260924 18:59:59.678519 2023 consensus_queue.cc:237] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } }
I20260924 18:59:59.680233 2062 tablet_replica.cc:439] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9: TxnStatusTablet state changed. Reason: New leader bd36d282dbbe4a58a9895b149f77ace9. Latest consensus state: current_term: 1 leader_uuid: "bd36d282dbbe4a58a9895b149f77ace9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } } }
I20260924 18:59:59.680610 2062 tablet_replica.cc:442] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9: This TxnStatusTablet replica's current role is: LEADER
I20260924 18:59:59.681262 2064 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 18:59:59.681470 2064 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 18:59:59.682004 1510 catalog_manager.cc:5495] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd36d282dbbe4a58a9895b149f77ace9 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "bd36d282dbbe4a58a9895b149f77ace9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } health_report { overall_health: HEALTHY } } }
I20260924 18:59:59.706677 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:59.701131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42001 (local address 127.0.0.1:38612)
0924 18:59:59.701384 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:59.701407 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:59.701596 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:59.702060 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:59.702064 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:59.702075 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 18:59:59.702243 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:59.702252 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.703290 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.703297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:59.704236 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:59.704247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.705051 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.705056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:59.705230 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:59.705276 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 18:59:59.706144 (+ 868us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 18:59:59.706164 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 18:59:59.706271 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 18:59:59.706414 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 18:59:59.706681 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:59.701157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38612 (local address 127.0.97.3:42001)
0924 18:59:59.701337 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:59.701343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:59.701473 (+ 130us) server_negotiation.cc:407] Connection header received
0924 18:59:59.701813 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:59.701820 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:59.701906 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:59.702041 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 18:59:59.702380 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.703164 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.704395 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.704872 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.705024 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:59.705462 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 18:59:59.706030 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 18:59:59.706189 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:59.706290 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 18:59:59.706369 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 18:59:59.711424 1673 raft_consensus.cc:1261] T 7795765281244264856c4434a3af6fab P afb2353e1d634837966bb862c6e9753a [term 1 FOLLOWER]: Refusing update from remote peer bd36d282dbbe4a58a9895b149f77ace9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 18:59:59.711750 1812 raft_consensus.cc:1261] T 7795765281244264856c4434a3af6fab P a402da4abc2a40309caa3ab7663c093a [term 1 FOLLOWER]: Refusing update from remote peer bd36d282dbbe4a58a9895b149f77ace9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 18:59:59.712131 2062 consensus_queue.cc:1035] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 18:59:59.712877 2023 consensus_queue.cc:1035] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 18:59:59.715945 2021 tablet_replica.cc:439] T 7795765281244264856c4434a3af6fab P afb2353e1d634837966bb862c6e9753a: TxnStatusTablet state changed. Reason: New leader bd36d282dbbe4a58a9895b149f77ace9. Latest consensus state: current_term: 1 leader_uuid: "bd36d282dbbe4a58a9895b149f77ace9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } } }
I20260924 18:59:59.716223 2021 tablet_replica.cc:442] T 7795765281244264856c4434a3af6fab P afb2353e1d634837966bb862c6e9753a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 18:59:59.718207 2068 mvcc.cc:205] Tried to move back new op lower bound from 7332972133211684864 to 7332972133086580736. Current Snapshot: MvccSnapshot[applied={T|T < 7332972133211684864}]
I20260924 18:59:59.719628 2067 mvcc.cc:205] Tried to move back new op lower bound from 7332972133211684864 to 7332972133086580736. Current Snapshot: MvccSnapshot[applied={T|T < 7332972133211684864}]
I20260924 18:59:59.725948 2021 tablet_replica.cc:439] T 7795765281244264856c4434a3af6fab P afb2353e1d634837966bb862c6e9753a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd36d282dbbe4a58a9895b149f77ace9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } } }
I20260924 18:59:59.726220 2021 tablet_replica.cc:442] T 7795765281244264856c4434a3af6fab P afb2353e1d634837966bb862c6e9753a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 18:59:59.728587 2022 tablet_replica.cc:439] T 7795765281244264856c4434a3af6fab P a402da4abc2a40309caa3ab7663c093a: TxnStatusTablet state changed. Reason: New leader bd36d282dbbe4a58a9895b149f77ace9. Latest consensus state: current_term: 1 leader_uuid: "bd36d282dbbe4a58a9895b149f77ace9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } } }
I20260924 18:59:59.728916 2022 tablet_replica.cc:442] T 7795765281244264856c4434a3af6fab P a402da4abc2a40309caa3ab7663c093a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 18:59:59.731601 2023 tablet_replica.cc:439] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd36d282dbbe4a58a9895b149f77ace9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } } }
I20260924 18:59:59.731969 2023 tablet_replica.cc:442] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9: This TxnStatusTablet replica's current role is: LEADER
I20260924 18:59:59.733180 2023 tablet_replica.cc:439] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd36d282dbbe4a58a9895b149f77ace9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } } }
I20260924 18:59:59.733484 2023 tablet_replica.cc:442] T 7795765281244264856c4434a3af6fab P bd36d282dbbe4a58a9895b149f77ace9: This TxnStatusTablet replica's current role is: LEADER
I20260924 18:59:59.737934 2022 tablet_replica.cc:439] T 7795765281244264856c4434a3af6fab P a402da4abc2a40309caa3ab7663c093a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd36d282dbbe4a58a9895b149f77ace9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb2353e1d634837966bb862c6e9753a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42503 } } peers { permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 } } peers { permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 } } }
I20260924 18:59:59.738271 2022 tablet_replica.cc:442] T 7795765281244264856c4434a3af6fab P a402da4abc2a40309caa3ab7663c093a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 18:59:59.746680 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:59.740110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48794 (local address 127.0.97.1:42503)
0924 18:59:59.740299 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:59.740305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:59.740410 (+ 105us) server_negotiation.cc:407] Connection header received
0924 18:59:59.740695 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:59.740701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:59.740816 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:59.740973 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 18:59:59.741380 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.742519 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.744266 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.744877 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.745008 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:59.745355 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 18:59:59.746126 (+ 771us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 18:59:59.746277 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:59.746333 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 18:59:59.746411 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 18:59:59.765364 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:59.757917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54551 (local address 127.0.97.3:42001)
0924 18:59:59.758097 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:59.758104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:59.758674 (+ 570us) server_negotiation.cc:407] Connection header received
0924 18:59:59.758794 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:59.758800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:59.758895 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:59.759009 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 18:59:59.759661 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.760822 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.762873 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.763600 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.763721 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:59.764634 (+ 913us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 18:59:59.764776 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 18:59:59.764867 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:59.765016 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 18:59:59.765089 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 18:59:59.765523 2079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:59.757900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42001 (local address 127.0.97.1:54551)
0924 18:59:59.758535 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:59.758563 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:59.758691 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:59.759168 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:59.759173 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:59.759185 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 18:59:59.759502 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:59.759513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.760960 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.760966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:59.762728 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:59.762743 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.763779 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.763786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:59.764394 (+ 608us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:59.764461 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 18:59:59.764888 (+ 427us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 18:59:59.764899 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 18:59:59.764981 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 18:59:59.765124 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":349,"threads_started":1}
I20260924 18:59:59.774657 1812 raft_consensus.cc:1261] T 78fa364e04e34a54ad9393d7efa9fed2 P a402da4abc2a40309caa3ab7663c093a [term 1 FOLLOWER]: Refusing update from remote peer afb2353e1d634837966bb862c6e9753a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 18:59:59.774673 1946 raft_consensus.cc:1261] T 78fa364e04e34a54ad9393d7efa9fed2 P bd36d282dbbe4a58a9895b149f77ace9 [term 1 FOLLOWER]: Refusing update from remote peer afb2353e1d634837966bb862c6e9753a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 18:59:59.775406 2047 consensus_queue.cc:1035] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd36d282dbbe4a58a9895b149f77ace9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 18:59:59.776139 2021 consensus_queue.cc:1035] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a402da4abc2a40309caa3ab7663c093a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 18:59:59.832726 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:59.825168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33347 (local address 127.0.97.1:42503)
0924 18:59:59.825422 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:59.825429 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:59.826005 (+ 576us) server_negotiation.cc:407] Connection header received
0924 18:59:59.826264 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:59.826270 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:59.826376 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:59.826441 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 18:59:59.827193 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.828346 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.830546 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.831219 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.831312 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:59.831956 (+ 644us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 18:59:59.832091 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 18:59:59.832194 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:59.832340 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 18:59:59.832418 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260924 18:59:59.833052 2086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:59.824506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42503 (local address 127.0.97.3:33347)
0924 18:59:59.825912 (+ 1406us) negotiation.cc:107] Waiting for socket to connect
0924 18:59:59.825931 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 18:59:59.826116 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 18:59:59.826620 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 18:59:59.826628 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 18:59:59.826643 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 18:59:59.827022 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:59.827037 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.828512 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.828517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:59.830385 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 18:59:59.830399 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.831498 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.831504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 18:59:59.831677 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:59.831753 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 18:59:59.832209 (+ 456us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 18:59:59.832219 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 18:59:59.832298 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 18:59:59.832441 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1207,"thread_start_us":2697,"threads_started":1}
I20260924 18:59:59.875700 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 18:59:59.875942 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 18:59:59.879316 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24931820b37e4f298ea207374672357e: Sending DeleteTablet for 3 replicas of tablet 78fa364e04e34a54ad9393d7efa9fed2
I20260924 18:59:59.880052 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 78fa364e04e34a54ad9393d7efa9fed2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 18:59:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48784
I20260924 18:59:59.880336 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 78fa364e04e34a54ad9393d7efa9fed2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 18:59:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44350
I20260924 18:59:59.880901 1925 tablet_service.cc:1514] Processing DeleteTablet for tablet 78fa364e04e34a54ad9393d7efa9fed2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 18:59:59 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38608
I20260924 18:59:59.892983 2090 tablet_replica.cc:330] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a: stopping tablet replica
I20260924 18:59:59.893307 2090 raft_consensus.cc:2229] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a [term 1 LEADER]: Raft consensus shutting down.
I20260924 18:59:59.894042 2090 raft_consensus.cc:2258] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 18:59:59.895211 2090 ts_tablet_manager.cc:1905] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 18:59:59.903245 2090 ts_tablet_manager.cc:1918] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 18:59:59.903466 2090 log.cc:1196] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-0/wal/wals/78fa364e04e34a54ad9393d7efa9fed2
I20260924 18:59:59.904019 2090 ts_tablet_manager.cc:1939] T 78fa364e04e34a54ad9393d7efa9fed2 P afb2353e1d634837966bb862c6e9753a: Deleting consensus metadata
I20260924 18:59:59.905154 1497 catalog_manager.cc:4841] TS afb2353e1d634837966bb862c6e9753a (127.0.97.1:42503): tablet 78fa364e04e34a54ad9393d7efa9fed2 (table test-table [id=4b75e9e1d12c42a0b928e73190c1f614]) successfully deleted
I20260924 18:59:59.906234 2092 tablet_replica.cc:330] T 78fa364e04e34a54ad9393d7efa9fed2 P bd36d282dbbe4a58a9895b149f77ace9: stopping tablet replica
I20260924 18:59:59.906538 2092 raft_consensus.cc:2229] T 78fa364e04e34a54ad9393d7efa9fed2 P bd36d282dbbe4a58a9895b149f77ace9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 18:59:59.908028 2092 raft_consensus.cc:2258] T 78fa364e04e34a54ad9393d7efa9fed2 P bd36d282dbbe4a58a9895b149f77ace9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 18:59:59.908589 2091 tablet_replica.cc:330] T 78fa364e04e34a54ad9393d7efa9fed2 P a402da4abc2a40309caa3ab7663c093a: stopping tablet replica
I20260924 18:59:59.909039 2091 raft_consensus.cc:2229] T 78fa364e04e34a54ad9393d7efa9fed2 P a402da4abc2a40309caa3ab7663c093a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 18:59:59.909842 2091 raft_consensus.cc:2258] T 78fa364e04e34a54ad9393d7efa9fed2 P a402da4abc2a40309caa3ab7663c093a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 18:59:59.909911 2092 ts_tablet_manager.cc:1905] T 78fa364e04e34a54ad9393d7efa9fed2 P bd36d282dbbe4a58a9895b149f77ace9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 18:59:59.910923 2091 ts_tablet_manager.cc:1905] T 78fa364e04e34a54ad9393d7efa9fed2 P a402da4abc2a40309caa3ab7663c093a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 18:59:59.917798 2091 ts_tablet_manager.cc:1918] T 78fa364e04e34a54ad9393d7efa9fed2 P a402da4abc2a40309caa3ab7663c093a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 18:59:59.917981 2091 log.cc:1196] T 78fa364e04e34a54ad9393d7efa9fed2 P a402da4abc2a40309caa3ab7663c093a: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-1/wal/wals/78fa364e04e34a54ad9393d7efa9fed2
I20260924 18:59:59.918514 2091 ts_tablet_manager.cc:1939] T 78fa364e04e34a54ad9393d7efa9fed2 P a402da4abc2a40309caa3ab7663c093a: Deleting consensus metadata
I20260924 18:59:59.919586 1496 catalog_manager.cc:4841] TS a402da4abc2a40309caa3ab7663c093a (127.0.97.2:46661): tablet 78fa364e04e34a54ad9393d7efa9fed2 (table test-table [id=4b75e9e1d12c42a0b928e73190c1f614]) successfully deleted
I20260924 18:59:59.923913 2092 ts_tablet_manager.cc:1918] T 78fa364e04e34a54ad9393d7efa9fed2 P bd36d282dbbe4a58a9895b149f77ace9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 18:59:59.924114 2092 log.cc:1196] T 78fa364e04e34a54ad9393d7efa9fed2 P bd36d282dbbe4a58a9895b149f77ace9: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276386554690-388-0/minicluster-data/ts-2/wal/wals/78fa364e04e34a54ad9393d7efa9fed2
I20260924 18:59:59.924734 2092 ts_tablet_manager.cc:1939] T 78fa364e04e34a54ad9393d7efa9fed2 P bd36d282dbbe4a58a9895b149f77ace9: Deleting consensus metadata
I20260924 18:59:59.925740 1496 catalog_manager.cc:4841] TS bd36d282dbbe4a58a9895b149f77ace9 (127.0.97.3:42001): tablet 78fa364e04e34a54ad9393d7efa9fed2 (table test-table [id=4b75e9e1d12c42a0b928e73190c1f614]) successfully deleted
Sep 24 18:59:59 dist-test-slave-9fn1 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276398, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 18:59:59.940855 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:59.906507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48800 (local address 127.0.97.1:42503)
0924 18:59:59.906719 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:59.906725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:59.919390 (+ 12665us) server_negotiation.cc:407] Connection header received
0924 18:59:59.919711 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:59.919716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:59.919800 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:59.919865 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:59.922336 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.923563 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.925867 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.926591 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.926697 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:59.932837 (+ 6140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:59.932856 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:59.932880 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:59.932930 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:59.937306 (+ 4376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:59.938186 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:59.938197 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:59.938201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:59.938361 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:59.938785 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:59.938790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:59.938794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:59.939350 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:59.939671 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:59.940055 (+ 384us) server_negotiation.cc:299] Negotiation successful
0924 18:59:59.940294 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 18:59:59.941656 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48800
I20260924 18:59:59.962157 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:59.945574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39404 (local address 127.0.97.62:41805)
0924 18:59:59.945775 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:59.945781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:59.946252 (+ 471us) server_negotiation.cc:407] Connection header received
0924 18:59:59.946394 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:59.946399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:59.946478 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:59.946585 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:59.948300 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.949380 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.951338 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.952441 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.952564 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:59.955490 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:59.955497 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:59.955502 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:59.955543 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:59.959211 (+ 3668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:59.959975 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:59.959980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:59.959984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:59.960136 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:59.960534 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:59.960540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:59.960543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:59.960886 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:59.961137 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:59.961491 (+ 354us) server_negotiation.cc:299] Negotiation successful
0924 18:59:59.961778 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260924 18:59:59.962733 1511 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39404
I20260924 18:59:59.965370 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 1582
I20260924 18:59:59.989794 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 18:59:59.966465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48814 (local address 127.0.97.1:42503)
0924 18:59:59.967700 (+ 1235us) server_negotiation.cc:206] Beginning negotiation
0924 18:59:59.967706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 18:59:59.973395 (+ 5689us) server_negotiation.cc:407] Connection header received
0924 18:59:59.973613 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 18:59:59.973620 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 18:59:59.973714 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 18:59:59.973816 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 18:59:59.975417 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.976604 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.978562 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 18:59:59.979601 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 18:59:59.979734 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 18:59:59.982906 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 18:59:59.982913 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 18:59:59.982918 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 18:59:59.982956 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 18:59:59.986578 (+ 3622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:59.987371 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:59.987381 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:59.987385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:59.987539 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 18:59:59.987984 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 18:59:59.987990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 18:59:59.987993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 18:59:59.988349 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 18:59:59.988580 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 18:59:59.989150 (+ 570us) server_negotiation.cc:299] Negotiation successful
0924 18:59:59.989372 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1103}
W20260924 18:59:59.990396 1713 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48814
W20260924 18:59:59.990937 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 1582: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 18:59:59.991130 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 1582
I20260924 19:00:00.010057 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 1722
Sep 24 19:00:00 dist-test-slave-9fn1 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276398, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 19:00:00.025807 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:00.010860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44368 (local address 127.0.97.2:46661)
0924 19:00:00.011053 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:00.011061 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:00.011081 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:00:00.011286 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:00.011290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:00.011359 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:00.011464 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:00.012543 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:00.013251 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:00.014804 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:00.015214 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:00.015366 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:00.019537 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:00.019551 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:00.019565 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:00.019603 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:00.022598 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:00.023353 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:00.023365 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:00.023368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:00.023517 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:00.023954 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:00.023960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:00.023965 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:00.024469 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:00.024843 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:00.025135 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 19:00:00.025390 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 19:00:00.026484 1852 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44368
W20260924 19:00:00.027138 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 1722: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:00:00.027329 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 1722
I20260924 19:00:00.048717 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 1861
Sep 24 19:00:00 dist-test-slave-9fn1 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276398, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 19:00:00.065701 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:00.049556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38626 (local address 127.0.97.3:42001)
0924 19:00:00.049721 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:00.049727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:00.049798 (+ 71us) server_negotiation.cc:407] Connection header received
0924 19:00:00.049995 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:00.050004 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:00.050077 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:00.050187 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:00.051390 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:00.052152 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:00.053415 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:00.054253 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:00.054397 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:00.058830 (+ 4433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:00.058846 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:00.058871 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:00.058914 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:00.062834 (+ 3920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:00.063499 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:00.063509 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:00.063512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:00.063690 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:00.064040 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:00.064051 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:00.064054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:00.064504 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:00.064859 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:00.065015 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 19:00:00.065258 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260924 19:00:00.066396 1991 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38626
W20260924 19:00:00.067118 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 1861: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:00:00.067237 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 1861
I20260924 19:00:00.087071 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 1479
I20260924 19:00:00.088235 1551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:00.199710 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 1479
2026-09-24T19:00:00Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4701 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:00:00 dist-test-slave-9fn1 krb5kdc[2101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:00:00 dist-test-slave-9fn1 krb5kdc[2101](info): set up 2 sockets
Sep 24 19:00:00 dist-test-slave-9fn1 krb5kdc[2101](info): commencing operation
krb5kdc: starting...
W20260924 19:00:01.277413 388 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:00:01 dist-test-slave-9fn1 krb5kdc[2101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:00:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:00:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 19:00:01.455088 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33983
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:01.764638 2114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:01.764914 2114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:01.765070 2114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:01.772267 2114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:00:01.772382 2114 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 19:00:01.772437 2114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:01.772526 2114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:01.772601 2114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:00:01.772711 2114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:00:01.782459 2114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33983
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:01.783701 2114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:01.785010 2114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:01.796597 2119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:01.796530 2122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:01.796571 2120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:01.797542 2114 server_base.cc:1034] running on GCE node
I20260924 19:00:01.798372 2114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:01.799727 2114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:01.800978 2114 hybrid_clock.cc:648] HybridClock initialized: now 1790276401800905 us; error 52 us; skew 500 ppm
I20260924 19:00:01.801308 2114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:01 dist-test-slave-9fn1 krb5kdc[2101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:01.804301 2114 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:00:01.805723 2114 webserver.cc:460] Webserver started at http://127.0.97.62:37037/ using document root <none> and password file <none>
I20260924 19:00:01.806365 2114 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:01.806450 2114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:01.806700 2114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:01.809465 2114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "edba9d3537dc4d52b49c6117e3562a89"
format_stamp: "Formatted at 2026-09-24 19:00:01 on dist-test-slave-9fn1"
server_key: "ddc397d25a3b07399f36e636d94c33fb"
server_key_iv: "5ebf5e710827fb616974af663c3a41d3"
server_key_version: "encryptionkey@0"
I20260924 19:00:01.810324 2114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "edba9d3537dc4d52b49c6117e3562a89"
format_stamp: "Formatted at 2026-09-24 19:00:01 on dist-test-slave-9fn1"
server_key: "ddc397d25a3b07399f36e636d94c33fb"
server_key_iv: "5ebf5e710827fb616974af663c3a41d3"
server_key_version: "encryptionkey@0"
I20260924 19:00:01.815644 2114 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:00:01.818873 2129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:01.820143 2114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:00:01.820446 2114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "edba9d3537dc4d52b49c6117e3562a89"
format_stamp: "Formatted at 2026-09-24 19:00:01 on dist-test-slave-9fn1"
server_key: "ddc397d25a3b07399f36e636d94c33fb"
server_key_iv: "5ebf5e710827fb616974af663c3a41d3"
server_key_version: "encryptionkey@0"
I20260924 19:00:01.820653 2114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:01.848555 2114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:01.852484 2114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:01.852730 2114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:01.875461 2114 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33983
I20260924 19:00:01.875483 2190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33983 every 8 connection(s)
I20260924 19:00:01.876772 2114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:00:01.880360 2191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:01.881695 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 2114
I20260924 19:00:01.881891 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:00:01.882498 388 external_mini_cluster.cc:1428] Setting key f7e9bdf870112d13b51ccc1cf36619d1
I20260924 19:00:01.892020 2191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edba9d3537dc4d52b49c6117e3562a89: Bootstrap starting.
I20260924 19:00:01.895740 2191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edba9d3537dc4d52b49c6117e3562a89: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:01.896970 2191 log.cc:824] T 00000000000000000000000000000000 P edba9d3537dc4d52b49c6117e3562a89: Log is configured to *not* fsync() on all Append() calls
Sep 24 19:00:01 dist-test-slave-9fn1 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:01.899938 2191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edba9d3537dc4d52b49c6117e3562a89: No bootstrap required, opened a new log
I20260924 19:00:01.905316 2194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:01.884425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39730 (local address 127.0.97.62:33983)
0924 19:00:01.885269 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:01.885283 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:01.885325 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:00:01.887242 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:01.887263 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:01.887633 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:01.887941 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:01.889727 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:01.891037 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:01.892712 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:01.893352 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:01.893489 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:01.898184 (+ 4695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:01.898196 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:01.898216 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:01.898263 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:01.901303 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:01.902112 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:01.902118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:01.902127 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:01.902302 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:01.902716 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:01.902720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:01.902722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:01.903217 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:01.903474 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:01.903873 (+ 399us) server_negotiation.cc:299] Negotiation successful
0924 19:00:01.904302 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":390,"threads_started":1}
I20260924 19:00:01.908350 2191 raft_consensus.cc:348] T 00000000000000000000000000000000 P edba9d3537dc4d52b49c6117e3562a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edba9d3537dc4d52b49c6117e3562a89" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33983 } }
I20260924 19:00:01.908694 2191 raft_consensus.cc:374] T 00000000000000000000000000000000 P edba9d3537dc4d52b49c6117e3562a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:01.908813 2191 raft_consensus.cc:729] T 00000000000000000000000000000000 P edba9d3537dc4d52b49c6117e3562a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edba9d3537dc4d52b49c6117e3562a89, State: Initialized, Role: FOLLOWER
I20260924 19:00:01.909340 2191 consensus_queue.cc:260] T 00000000000000000000000000000000 P edba9d3537dc4d52b49c6117e3562a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edba9d3537dc4d52b49c6117e3562a89" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33983 } }
I20260924 19:00:01.909590 2191 raft_consensus.cc:388] T 00000000000000000000000000000000 P edba9d3537dc4d52b49c6117e3562a89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:00:01.909746 2191 raft_consensus.cc:482] T 00000000000000000000000000000000 P edba9d3537dc4d52b49c6117e3562a89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:00:01.909915 2191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edba9d3537dc4d52b49c6117e3562a89 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:01.912035 2191 raft_consensus.cc:504] T 00000000000000000000000000000000 P edba9d3537dc4d52b49c6117e3562a89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edba9d3537dc4d52b49c6117e3562a89" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33983 } }
I20260924 19:00:01.912602 2191 leader_election.cc:302] T 00000000000000000000000000000000 P edba9d3537dc4d52b49c6117e3562a89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edba9d3537dc4d52b49c6117e3562a89; no voters:
I20260924 19:00:01.913203 2191 leader_election.cc:288] T 00000000000000000000000000000000 P edba9d3537dc4d52b49c6117e3562a89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:00:01.913354 2196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edba9d3537dc4d52b49c6117e3562a89 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:00:01.914091 2196 raft_consensus.cc:686] T 00000000000000000000000000000000 P edba9d3537dc4d52b49c6117e3562a89 [term 1 LEADER]: Becoming Leader. State: Replica: edba9d3537dc4d52b49c6117e3562a89, State: Running, Role: LEADER
I20260924 19:00:01.914605 2196 consensus_queue.cc:237] T 00000000000000000000000000000000 P edba9d3537dc4d52b49c6117e3562a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edba9d3537dc4d52b49c6117e3562a89" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33983 } }
I20260924 19:00:01.915196 2191 sys_catalog.cc:564] T 00000000000000000000000000000000 P edba9d3537dc4d52b49c6117e3562a89 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:00:01.917317 2198 sys_catalog.cc:455] T 00000000000000000000000000000000 P edba9d3537dc4d52b49c6117e3562a89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edba9d3537dc4d52b49c6117e3562a89. Latest consensus state: current_term: 1 leader_uuid: "edba9d3537dc4d52b49c6117e3562a89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edba9d3537dc4d52b49c6117e3562a89" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33983 } } }
I20260924 19:00:01.917804 2198 sys_catalog.cc:458] T 00000000000000000000000000000000 P edba9d3537dc4d52b49c6117e3562a89 [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:01.918115 2197 sys_catalog.cc:455] T 00000000000000000000000000000000 P edba9d3537dc4d52b49c6117e3562a89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edba9d3537dc4d52b49c6117e3562a89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edba9d3537dc4d52b49c6117e3562a89" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33983 } } }
I20260924 19:00:01.918411 2197 sys_catalog.cc:458] T 00000000000000000000000000000000 P edba9d3537dc4d52b49c6117e3562a89 [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:01.920024 2204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:00:01.925820 2204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:00:01.934469 2204 catalog_manager.cc:1347] Generated new cluster ID: fe2db4699ede4adaad46676babebec34
I20260924 19:00:01.934587 2204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:00:01.953588 2204 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:00:01.954859 2204 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:00:01.970597 2204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edba9d3537dc4d52b49c6117e3562a89: Generated new TSK 0
I20260924 19:00:01.971419 2204 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 19:00:02.017556 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33983
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:02.336615 2217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:02.336901 2217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:02.337062 2217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:02.344381 2217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:02.344480 2217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:02.344663 2217 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 19:00:02.344726 2217 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:00:02.344821 2217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:00:02.354597 2217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:02.355877 2217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:02.357312 2217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:02.370162 2223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:02.370223 2225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:02.370240 2222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:02.371006 2217 server_base.cc:1034] running on GCE node
I20260924 19:00:02.371536 2217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:02.372367 2217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:02.373615 2217 hybrid_clock.cc:648] HybridClock initialized: now 1790276402373569 us; error 31 us; skew 500 ppm
I20260924 19:00:02.373947 2217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:02 dist-test-slave-9fn1 krb5kdc[2101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:02.377208 2217 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 19:00:02.378603 2217 webserver.cc:460] Webserver started at http://127.0.97.1:43657/ using document root <none> and password file <none>
I20260924 19:00:02.379225 2217 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:02.379323 2217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:02.379563 2217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:02.382459 2217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-0/data/instance:
uuid: "f819faf5f5fa4b339163667ef2d94ff4"
format_stamp: "Formatted at 2026-09-24 19:00:02 on dist-test-slave-9fn1"
server_key: "0c9a90b4200d167bbad9cba253268373"
server_key_iv: "9e753a6a10a9a03bf07458dbfba65c50"
server_key_version: "encryptionkey@0"
I20260924 19:00:02.383169 2217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f819faf5f5fa4b339163667ef2d94ff4"
format_stamp: "Formatted at 2026-09-24 19:00:02 on dist-test-slave-9fn1"
server_key: "0c9a90b4200d167bbad9cba253268373"
server_key_iv: "9e753a6a10a9a03bf07458dbfba65c50"
server_key_version: "encryptionkey@0"
I20260924 19:00:02.388020 2217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:00:02.391052 2232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:02.392238 2217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 19:00:02.392429 2217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "f819faf5f5fa4b339163667ef2d94ff4"
format_stamp: "Formatted at 2026-09-24 19:00:02 on dist-test-slave-9fn1"
server_key: "0c9a90b4200d167bbad9cba253268373"
server_key_iv: "9e753a6a10a9a03bf07458dbfba65c50"
server_key_version: "encryptionkey@0"
I20260924 19:00:02.392608 2217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:02.426391 2217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:02.430338 2217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:02.430632 2217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:02.433054 2217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:02.433178 2217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:02.433300 2217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:02.433382 2217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:00:02 dist-test-slave-9fn1 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:02.457526 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:02.433899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39387 (local address 127.0.97.62:33983)
0924 19:00:02.434351 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:02.434355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:02.435455 (+ 1100us) server_negotiation.cc:407] Connection header received
0924 19:00:02.437673 (+ 2218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:02.437679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:02.437770 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:02.437895 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:02.440571 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:02.441540 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:02.444731 (+ 3191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:02.445293 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:02.445424 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:02.450485 (+ 5061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:02.450495 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:02.450504 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:02.450555 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:02.454188 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:02.454894 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:02.454899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:02.454903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:02.455033 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:02.455466 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:02.455472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:02.455475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:02.455738 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:02.455888 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:02.456664 (+ 776us) server_negotiation.cc:299] Negotiation successful
0924 19:00:02.456963 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":255,"threads_started":1}
I20260924 19:00:02.458359 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:02.434535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33983 (local address 127.0.97.1:39387)
0924 19:00:02.435298 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:02.435342 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:02.437341 (+ 1999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:02.438332 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:02.438346 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:02.439284 (+ 938us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:02.440328 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:02.440366 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:02.441714 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:02.441720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:02.444550 (+ 2830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:02.444573 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:02.445766 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:02.445776 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:02.446121 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:02.446871 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:02.446912 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:02.450227 (+ 3315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:02.454327 (+ 4100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:02.454335 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:02.454351 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:02.454763 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:02.455142 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:02.455148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:02.455152 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:02.455354 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:02.455865 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:02.455873 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:02.456230 (+ 357us) client_negotiation.cc:769] Sending connection context
0924 19:00:02.456575 (+ 345us) client_negotiation.cc:240] Negotiation successful
0924 19:00:02.457003 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":330,"threads_started":1}
I20260924 19:00:02.478691 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:02.464638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41107 (local address 127.0.97.62:33983)
0924 19:00:02.464853 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:02.464861 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:02.465540 (+ 679us) server_negotiation.cc:407] Connection header received
0924 19:00:02.465742 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:02.465747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:02.465816 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:02.465911 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:02.467320 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:02.468365 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:02.470217 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:02.470615 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:02.470742 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:02.472508 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:02.472521 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:02.472525 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:02.472552 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:02.475691 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:02.476698 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:02.476702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:02.476704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:02.476820 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:02.477289 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:02.477294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:02.477298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:02.477626 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:02.477787 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:02.478302 (+ 515us) server_negotiation.cc:299] Negotiation successful
0924 19:00:02.478452 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 19:00:02.479267 2317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:02.464692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33983 (local address 127.0.97.1:41107)
0924 19:00:02.465400 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:02.465423 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:02.465611 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:02.465932 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:02.465937 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:02.466625 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:02.467110 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:02.467127 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:02.468532 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:02.468537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:02.469993 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:02.470033 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:02.470806 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:02.470813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:02.470992 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:02.471634 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:02.471650 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:02.472337 (+ 687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:02.475844 (+ 3507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:02.475852 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:02.475857 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:02.476523 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:02.476989 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:02.476997 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:02.477001 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:02.477171 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:02.477763 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:02.477770 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:02.478030 (+ 260us) client_negotiation.cc:769] Sending connection context
0924 19:00:02.478219 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 19:00:02.478633 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":394,"threads_started":1}
I20260924 19:00:02.487754 2217 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32905
I20260924 19:00:02.487830 2352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32905 every 8 connection(s)
I20260924 19:00:02.488940 2217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:00:02.490792 2353 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33983
I20260924 19:00:02.490968 2353 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:02.491479 2353 heartbeater.cc:511] Master 127.0.97.62:33983 requested a full tablet report, sending...
I20260924 19:00:02.493194 2146 ts_manager.cc:194] Registered new tserver with Master: f819faf5f5fa4b339163667ef2d94ff4 (127.0.97.1:32905)
I20260924 19:00:02.494588 2146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39387
I20260924 19:00:02.496912 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 2217
I20260924 19:00:02.497123 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-0/wal/instance
I20260924 19:00:02.497772 388 external_mini_cluster.cc:1428] Setting key 26b0ba9e0a273c5190f3e188790ca959
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 19:00:02.536464 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33983
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:02.845325 2357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:02.845676 2357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:02.845865 2357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:02.852898 2357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:02.853041 2357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:02.853202 2357 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 19:00:02.853292 2357 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:00:02.853398 2357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:00:02.863085 2357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:02.864318 2357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:02.865674 2357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:02.879086 2362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:02.879635 2365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:02.879738 2363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:02.880527 2357 server_base.cc:1034] running on GCE node
I20260924 19:00:02.881181 2357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:02.882097 2357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:02.883343 2357 hybrid_clock.cc:648] HybridClock initialized: now 1790276402883279 us; error 52 us; skew 500 ppm
I20260924 19:00:02.883690 2357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:02 dist-test-slave-9fn1 krb5kdc[2101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:02.886912 2357 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 19:00:02.892194 2357 webserver.cc:460] Webserver started at http://127.0.97.2:32775/ using document root <none> and password file <none>
I20260924 19:00:02.892899 2357 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:02.892995 2357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:02.893345 2357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:02.896265 2357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-1/data/instance:
uuid: "f622b4e172b6419eb1ae8033dcbd8c41"
format_stamp: "Formatted at 2026-09-24 19:00:02 on dist-test-slave-9fn1"
server_key: "39e609d6da4828b590aeeb7493048d67"
server_key_iv: "35075f560b4d47d5961b826f5b410f73"
server_key_version: "encryptionkey@0"
I20260924 19:00:02.897070 2357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f622b4e172b6419eb1ae8033dcbd8c41"
format_stamp: "Formatted at 2026-09-24 19:00:02 on dist-test-slave-9fn1"
server_key: "39e609d6da4828b590aeeb7493048d67"
server_key_iv: "35075f560b4d47d5961b826f5b410f73"
server_key_version: "encryptionkey@0"
I20260924 19:00:02.902499 2357 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260924 19:00:02.905915 2372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:02.907178 2357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 19:00:02.907344 2357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "f622b4e172b6419eb1ae8033dcbd8c41"
format_stamp: "Formatted at 2026-09-24 19:00:02 on dist-test-slave-9fn1"
server_key: "39e609d6da4828b590aeeb7493048d67"
server_key_iv: "35075f560b4d47d5961b826f5b410f73"
server_key_version: "encryptionkey@0"
I20260924 19:00:02.907519 2357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:02.938421 2357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:02.942452 2357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:02.942759 2357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:02.945186 2357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:02.945343 2357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:02.945472 2357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:02.945582 2357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 24 19:00:02 dist-test-slave-9fn1 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:02.971858 2381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:02.946644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33983 (local address 127.0.97.2:52991)
0924 19:00:02.947449 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:02.947507 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:02.950447 (+ 2940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:02.951339 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:02.951357 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:02.952320 (+ 963us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:02.953346 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:02.953379 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:02.954707 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:02.954719 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:02.957557 (+ 2838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:02.957577 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:02.958604 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:02.958622 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:02.958827 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:02.959663 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:02.959705 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:02.963133 (+ 3428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:02.967295 (+ 4162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:02.967307 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:02.967329 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:02.967813 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:02.968225 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:02.968230 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:02.968233 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:02.968425 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:02.969135 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:02.969146 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:02.969643 (+ 497us) client_negotiation.cc:769] Sending connection context
0924 19:00:02.970003 (+ 360us) client_negotiation.cc:240] Negotiation successful
0924 19:00:02.970504 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":338,"threads_started":1}
I20260924 19:00:02.972728 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:02.945993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52991 (local address 127.0.97.62:33983)
0924 19:00:02.946167 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:02.946173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:02.947642 (+ 1469us) server_negotiation.cc:407] Connection header received
0924 19:00:02.950810 (+ 3168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:02.950818 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:02.950915 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:02.951028 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:02.953584 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:02.954542 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:02.957739 (+ 3197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:02.958260 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:02.958398 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:02.963435 (+ 5037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:02.963444 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:02.963450 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:02.963497 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:02.967111 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:02.967938 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:02.967946 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:02.967951 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:02.968107 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:02.968548 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:02.968556 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:02.968561 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:02.968968 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:02.969146 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:02.972363 (+ 3217us) server_negotiation.cc:299] Negotiation successful
0924 19:00:02.972515 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 19:00:02.991884 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:02.978172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56845 (local address 127.0.97.62:33983)
0924 19:00:02.978368 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:02.978376 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:02.978863 (+ 487us) server_negotiation.cc:407] Connection header received
0924 19:00:02.978994 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:02.979001 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:02.979108 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:02.979232 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:02.980153 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:02.981585 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:02.982846 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:02.983217 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:02.983356 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:02.985611 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:02.985617 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:02.985621 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:02.985652 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:02.988790 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:02.989707 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:02.989714 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:02.989720 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:02.989858 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:02.990292 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:02.990299 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:02.990304 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:02.990651 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:02.990819 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:02.991377 (+ 558us) server_negotiation.cc:299] Negotiation successful
0924 19:00:02.991568 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 19:00:02.992020 2464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:02.978152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33983 (local address 127.0.97.2:56845)
0924 19:00:02.978757 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:02.978786 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:02.978886 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:02.979243 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:02.979246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:02.979774 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:02.980030 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:02.980038 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:02.981733 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:02.981737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:02.982707 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:02.982717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:02.983396 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:02.983402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:02.983607 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:02.984350 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:02.984377 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:02.985316 (+ 939us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:02.988892 (+ 3576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:02.988899 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:02.988905 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:02.989489 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:02.989961 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:02.989968 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:02.989972 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:02.990136 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:02.990890 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:02.990897 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:02.991114 (+ 217us) client_negotiation.cc:769] Sending connection context
0924 19:00:02.991292 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 19:00:02.991585 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":333,"threads_started":1}
I20260924 19:00:02.999639 2357 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43977
I20260924 19:00:02.999712 2491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43977 every 8 connection(s)
I20260924 19:00:03.000811 2357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:00:03.002733 2492 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33983
I20260924 19:00:03.002942 2492 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:03.003338 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 2357
I20260924 19:00:03.003481 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-1/wal/instance
I20260924 19:00:03.003579 2492 heartbeater.cc:511] Master 127.0.97.62:33983 requested a full tablet report, sending...
I20260924 19:00:03.003925 388 external_mini_cluster.cc:1428] Setting key 13cc23fcf062029fba84c15eb92ea74d
I20260924 19:00:03.004875 2146 ts_manager.cc:194] Registered new tserver with Master: f622b4e172b6419eb1ae8033dcbd8c41 (127.0.97.2:43977)
I20260924 19:00:03.005851 2146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52991
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 19:00:03.040516 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33983
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:03.350030 2496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:03.350307 2496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:03.350469 2496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:03.357540 2496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:03.357642 2496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:03.357831 2496 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 19:00:03.357890 2496 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:00:03.358014 2496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:00:03.367744 2496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:33983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:03.368917 2496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:03.370218 2496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:03.382656 2502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:03.382694 2501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:03.382916 2504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:03.383548 2496 server_base.cc:1034] running on GCE node
I20260924 19:00:03.384058 2496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:03.384896 2496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:03.386117 2496 hybrid_clock.cc:648] HybridClock initialized: now 1790276403386062 us; error 45 us; skew 500 ppm
I20260924 19:00:03.386409 2496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:03 dist-test-slave-9fn1 krb5kdc[2101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:03.399111 2496 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 19:00:03.400645 2496 webserver.cc:460] Webserver started at http://127.0.97.3:43639/ using document root <none> and password file <none>
I20260924 19:00:03.401346 2496 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:03.401484 2496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:03.401856 2496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:03.404567 2496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-2/data/instance:
uuid: "9691585dd5b94d62aeab6b7bc1929fae"
format_stamp: "Formatted at 2026-09-24 19:00:03 on dist-test-slave-9fn1"
server_key: "1c14125918426393aa52b55cee1cf991"
server_key_iv: "94a6e0eb413d0d268390390a889dec33"
server_key_version: "encryptionkey@0"
I20260924 19:00:03.405447 2496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9691585dd5b94d62aeab6b7bc1929fae"
format_stamp: "Formatted at 2026-09-24 19:00:03 on dist-test-slave-9fn1"
server_key: "1c14125918426393aa52b55cee1cf991"
server_key_iv: "94a6e0eb413d0d268390390a889dec33"
server_key_version: "encryptionkey@0"
I20260924 19:00:03.410354 2496 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 19:00:03.413645 2511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:03.414932 2496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 19:00:03.415118 2496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "9691585dd5b94d62aeab6b7bc1929fae"
format_stamp: "Formatted at 2026-09-24 19:00:03 on dist-test-slave-9fn1"
server_key: "1c14125918426393aa52b55cee1cf991"
server_key_iv: "94a6e0eb413d0d268390390a889dec33"
server_key_version: "encryptionkey@0"
I20260924 19:00:03.415279 2496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:03.460168 2496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:03.464246 2496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:03.464538 2496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:03.467123 2496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:03.467208 2496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:03.467410 2496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:03.467518 2496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:00:03 dist-test-slave-9fn1 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:03.497180 2353 heartbeater.cc:503] Master 127.0.97.62:33983 was elected leader, sending a full tablet report...
I20260924 19:00:03.499257 2520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:03.468380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33983 (local address 127.0.97.3:59821)
0924 19:00:03.469251 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:03.469300 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:03.471309 (+ 2009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:03.472422 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:03.472438 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:03.473322 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:03.474478 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:03.474508 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.475735 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.475745 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:03.478691 (+ 2946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:03.478727 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.479785 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.479796 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:03.480173 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:03.480987 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:03.481024 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:03.484539 (+ 3515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:03.494336 (+ 9797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:03.494349 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:03.494370 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:03.495067 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:03.495449 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:03.495454 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:03.495458 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:03.495686 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:03.496739 (+ 1053us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:03.496745 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:03.497103 (+ 358us) client_negotiation.cc:769] Sending connection context
0924 19:00:03.497388 (+ 285us) client_negotiation.cc:240] Negotiation successful
0924 19:00:03.497931 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":352,"threads_started":1}
I20260924 19:00:03.500263 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:03.467893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59821 (local address 127.0.97.62:33983)
0924 19:00:03.468089 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:03.468095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:03.469439 (+ 1344us) server_negotiation.cc:407] Connection header received
0924 19:00:03.471670 (+ 2231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:03.471679 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:03.471798 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:03.471920 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:03.474688 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.475514 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.478884 (+ 3370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.479478 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.479621 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:03.484814 (+ 5193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:03.484822 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:03.484829 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:03.484876 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:03.494137 (+ 9261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:03.495201 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:03.495207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:03.495212 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:03.495335 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:03.495808 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:03.495812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:03.495814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:03.496151 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:03.496305 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:03.499861 (+ 3556us) server_negotiation.cc:299] Negotiation successful
0924 19:00:03.500036 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:00:03.516009 2496 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42637
I20260924 19:00:03.516045 2630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42637 every 8 connection(s)
I20260924 19:00:03.517673 2496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:00:03.520987 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:03.505223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58081 (local address 127.0.97.62:33983)
0924 19:00:03.505361 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:03.505364 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:03.506182 (+ 818us) server_negotiation.cc:407] Connection header received
0924 19:00:03.506395 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:03.506400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:03.506463 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:03.506560 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:03.507897 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.509174 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.511289 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.512158 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.512309 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:03.514312 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:03.514317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:03.514321 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:03.514354 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:03.518013 (+ 3659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:03.518956 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:03.518963 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:03.518967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:03.519119 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:03.519534 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:03.519537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:03.519539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:03.519875 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:03.520038 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:03.520558 (+ 520us) server_negotiation.cc:299] Negotiation successful
0924 19:00:03.520710 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 19:00:03.520082 2631 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33983
I20260924 19:00:03.520991 2607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:03.505318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33983 (local address 127.0.97.3:58081)
0924 19:00:03.506088 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:03.506103 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:03.506243 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:03.506589 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:03.506594 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:03.507324 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:03.507709 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:03.507721 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.509320 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.509327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:03.511120 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:03.511142 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.512402 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.512410 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:03.512524 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:03.513212 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:03.513228 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:03.514130 (+ 902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:03.518168 (+ 4038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:03.518178 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:03.518185 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:03.518801 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:03.519271 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:03.519275 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:03.519278 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:03.519428 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:03.520031 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:03.520038 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:03.520234 (+ 196us) client_negotiation.cc:769] Sending connection context
0924 19:00:03.520409 (+ 175us) client_negotiation.cc:240] Negotiation successful
0924 19:00:03.520630 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":473,"threads_started":1}
I20260924 19:00:03.521395 2631 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:03.522176 2631 heartbeater.cc:511] Master 127.0.97.62:33983 requested a full tablet report, sending...
I20260924 19:00:03.523679 2146 ts_manager.cc:194] Registered new tserver with Master: 9691585dd5b94d62aeab6b7bc1929fae (127.0.97.3:42637)
I20260924 19:00:03.524528 2146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:59821
I20260924 19:00:03.527256 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 2496
I20260924 19:00:03.527434 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-2/wal/instance
I20260924 19:00:03.527851 388 external_mini_cluster.cc:1428] Setting key 363e3873326849b980789f76c436d3bb
I20260924 19:00:03.530439 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 19:00:03 dist-test-slave-9fn1 krb5kdc[2101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 19:00:03 dist-test-slave-9fn1 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:03.575717 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:03.562812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39742 (local address 127.0.97.62:33983)
0924 19:00:03.563028 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:03.563034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:03.563187 (+ 153us) server_negotiation.cc:407] Connection header received
0924 19:00:03.563328 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:03.563332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:03.563393 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:03.563478 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:03.564725 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.565408 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.566593 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.567020 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.567158 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:03.570758 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:03.570772 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:03.570776 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:03.570806 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:03.573048 (+ 2242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:03.573703 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:03.573710 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:03.573714 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:03.573838 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:03.574267 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:03.574272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:03.574275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:03.574538 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:03.574802 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:03.575104 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 19:00:03.575354 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 19:00:03.579476 2146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:00:03.586115 2146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:00:03.608764 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:03.597546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43977 (local address 127.0.0.1:43378)
0924 19:00:03.598318 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:03.598358 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:03.598538 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:03.599952 (+ 1414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:03.599964 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:03.600006 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:03.600509 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:03.600526 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.603473 (+ 2947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.603482 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:03.605977 (+ 2495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:03.605988 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.607509 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.607517 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:03.607855 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:03.607934 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 19:00:03.608107 (+ 173us) client_negotiation.cc:240] Negotiation successful
0924 19:00:03.608304 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":37,"thread_start_us":352,"threads_started":1}
I20260924 19:00:03.609496 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:03.597578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43378 (local address 127.0.97.2:43977)
0924 19:00:03.599140 (+ 1562us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:03.599150 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:03.599177 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:00:03.599288 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:03.599300 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:03.599612 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:03.600846 (+ 1234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:03.601109 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.603295 (+ 2186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.606142 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.607295 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.607429 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:03.607635 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:03.609050 (+ 1415us) server_negotiation.cc:299] Negotiation successful
0924 19:00:03.609220 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1379,"thread_start_us":1217,"threads_started":1}
I20260924 19:00:03.612161 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:03.598747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32905 (local address 127.0.0.1:36486)
0924 19:00:03.604070 (+ 5323us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:03.604089 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:03.604195 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:03.604766 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:03.604770 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:03.604794 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:03.605260 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:03.605269 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.607137 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.607142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:03.609318 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:03.609334 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.611335 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.611341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:03.611549 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:03.611585 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 19:00:03.611674 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 19:00:03.611800 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5183,"mutex_wait_us":70,"spinlock_wait_cycles":1920,"thread_start_us":4945,"threads_started":1}
I20260924 19:00:03.612320 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:03.599223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36486 (local address 127.0.97.1:32905)
0924 19:00:03.600009 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:03.600020 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:03.604323 (+ 4303us) server_negotiation.cc:407] Connection header received
0924 19:00:03.604403 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:03.604408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:03.604589 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:03.604780 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:03.605426 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.606997 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.609485 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.610828 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.610982 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:03.611204 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:03.611751 (+ 547us) server_negotiation.cc:299] Negotiation successful
0924 19:00:03.611915 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":472,"threads_started":1}
I20260924 19:00:03.613317 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:03.597944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42637 (local address 127.0.0.1:41822)
0924 19:00:03.599361 (+ 1417us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:03.599382 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:03.599509 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:03.604926 (+ 5417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:03.604931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:03.604975 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:03.605431 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:03.605442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.607523 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.607527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:03.610722 (+ 3195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:03.610738 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.612492 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.612499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:03.612671 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:03.612705 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 19:00:03.612818 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 19:00:03.612942 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1302,"mutex_wait_us":49,"spinlock_wait_cycles":5760,"thread_start_us":1172,"threads_started":1}
I20260924 19:00:03.613365 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:03.598537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41822 (local address 127.0.97.3:42637)
0924 19:00:03.604349 (+ 5812us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:03.604356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:03.604388 (+ 32us) server_negotiation.cc:407] Connection header received
0924 19:00:03.604475 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:03.604481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:03.604702 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:03.605634 (+ 932us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:03.605941 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.607345 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.610896 (+ 3551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.612303 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.612436 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:03.612653 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:03.612856 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 19:00:03.613014 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5636,"thread_start_us":5461,"threads_started":1}
I20260924 19:00:03.613272 2422 tablet_service.cc:1467] Processing CreateTablet for tablet 95d79252fac4450985859064776b1513 (DEFAULT_TABLE table=test-table [id=9fed71d60b7f41df992d1f7757cc08a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:03.614513 2422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95d79252fac4450985859064776b1513. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:03.616453 2561 tablet_service.cc:1467] Processing CreateTablet for tablet 95d79252fac4450985859064776b1513 (DEFAULT_TABLE table=test-table [id=9fed71d60b7f41df992d1f7757cc08a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:03.617751 2561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95d79252fac4450985859064776b1513. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:03.620803 2283 tablet_service.cc:1467] Processing CreateTablet for tablet 95d79252fac4450985859064776b1513 (DEFAULT_TABLE table=test-table [id=9fed71d60b7f41df992d1f7757cc08a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:03.622121 2283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95d79252fac4450985859064776b1513. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:03.629304 2651 tablet_bootstrap.cc:492] T 95d79252fac4450985859064776b1513 P 9691585dd5b94d62aeab6b7bc1929fae: Bootstrap starting.
I20260924 19:00:03.632624 2651 tablet_bootstrap.cc:654] T 95d79252fac4450985859064776b1513 P 9691585dd5b94d62aeab6b7bc1929fae: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:03.633812 2651 log.cc:824] T 95d79252fac4450985859064776b1513 P 9691585dd5b94d62aeab6b7bc1929fae: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:03.634338 2650 tablet_bootstrap.cc:492] T 95d79252fac4450985859064776b1513 P f622b4e172b6419eb1ae8033dcbd8c41: Bootstrap starting.
I20260924 19:00:03.637665 2653 tablet_bootstrap.cc:492] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4: Bootstrap starting.
I20260924 19:00:03.637969 2650 tablet_bootstrap.cc:654] T 95d79252fac4450985859064776b1513 P f622b4e172b6419eb1ae8033dcbd8c41: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:03.638149 2651 tablet_bootstrap.cc:492] T 95d79252fac4450985859064776b1513 P 9691585dd5b94d62aeab6b7bc1929fae: No bootstrap required, opened a new log
I20260924 19:00:03.638435 2651 ts_tablet_manager.cc:1397] T 95d79252fac4450985859064776b1513 P 9691585dd5b94d62aeab6b7bc1929fae: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260924 19:00:03.639401 2650 log.cc:824] T 95d79252fac4450985859064776b1513 P f622b4e172b6419eb1ae8033dcbd8c41: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:03.640790 2653 tablet_bootstrap.cc:654] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:03.641844 2653 log.cc:824] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:03.642478 2650 tablet_bootstrap.cc:492] T 95d79252fac4450985859064776b1513 P f622b4e172b6419eb1ae8033dcbd8c41: No bootstrap required, opened a new log
I20260924 19:00:03.642743 2650 ts_tablet_manager.cc:1397] T 95d79252fac4450985859064776b1513 P f622b4e172b6419eb1ae8033dcbd8c41: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 19:00:03.644636 2653 tablet_bootstrap.cc:492] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4: No bootstrap required, opened a new log
I20260924 19:00:03.644876 2653 ts_tablet_manager.cc:1397] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260924 19:00:03.648772 2651 raft_consensus.cc:348] T 95d79252fac4450985859064776b1513 P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:03.649201 2651 raft_consensus.cc:374] T 95d79252fac4450985859064776b1513 P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:03.649299 2651 raft_consensus.cc:729] T 95d79252fac4450985859064776b1513 P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9691585dd5b94d62aeab6b7bc1929fae, State: Initialized, Role: FOLLOWER
I20260924 19:00:03.649940 2651 consensus_queue.cc:260] T 95d79252fac4450985859064776b1513 P 9691585dd5b94d62aeab6b7bc1929fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:03.651023 2631 heartbeater.cc:503] Master 127.0.97.62:33983 was elected leader, sending a full tablet report...
I20260924 19:00:03.651726 2651 ts_tablet_manager.cc:1428] T 95d79252fac4450985859064776b1513 P 9691585dd5b94d62aeab6b7bc1929fae: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260924 19:00:03.652426 2653 raft_consensus.cc:348] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:03.652846 2653 raft_consensus.cc:374] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:03.652964 2653 raft_consensus.cc:729] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f819faf5f5fa4b339163667ef2d94ff4, State: Initialized, Role: FOLLOWER
I20260924 19:00:03.652912 2650 raft_consensus.cc:348] T 95d79252fac4450985859064776b1513 P f622b4e172b6419eb1ae8033dcbd8c41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:03.653302 2650 raft_consensus.cc:374] T 95d79252fac4450985859064776b1513 P f622b4e172b6419eb1ae8033dcbd8c41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:03.653406 2650 raft_consensus.cc:729] T 95d79252fac4450985859064776b1513 P f622b4e172b6419eb1ae8033dcbd8c41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f622b4e172b6419eb1ae8033dcbd8c41, State: Initialized, Role: FOLLOWER
I20260924 19:00:03.653599 2653 consensus_queue.cc:260] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:03.653977 2650 consensus_queue.cc:260] T 95d79252fac4450985859064776b1513 P f622b4e172b6419eb1ae8033dcbd8c41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:03.656479 2492 heartbeater.cc:503] Master 127.0.97.62:33983 was elected leader, sending a full tablet report...
I20260924 19:00:03.656935 2650 ts_tablet_manager.cc:1428] T 95d79252fac4450985859064776b1513 P f622b4e172b6419eb1ae8033dcbd8c41: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260924 19:00:03.660591 2653 ts_tablet_manager.cc:1428] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4: Time spent starting tablet: real 0.016s user 0.016s sys 0.001s
W20260924 19:00:03.741588 2354 tablet.cc:2369] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:00:03.752844 2493 tablet.cc:2369] T 95d79252fac4450985859064776b1513 P f622b4e172b6419eb1ae8033dcbd8c41: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:00:03.763047 2657 raft_consensus.cc:482] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:00:03.763314 2657 raft_consensus.cc:504] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:03.765298 2657 leader_election.cc:288] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f622b4e172b6419eb1ae8033dcbd8c41 (127.0.97.2:43977), 9691585dd5b94d62aeab6b7bc1929fae (127.0.97.3:42637)
W20260924 19:00:03.769797 2632 tablet.cc:2369] T 95d79252fac4450985859064776b1513 P 9691585dd5b94d62aeab6b7bc1929fae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:00:03.772747 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:03.765729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50895 (local address 127.0.97.3:42637)
0924 19:00:03.765952 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:03.765958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:03.766656 (+ 698us) server_negotiation.cc:407] Connection header received
0924 19:00:03.766878 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:03.766883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:03.766957 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:03.767019 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:03.767801 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.768550 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.770444 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.771665 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.771807 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:03.771924 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:03.772374 (+ 450us) server_negotiation.cc:299] Negotiation successful
0924 19:00:03.772496 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 19:00:03.772747 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:03.765533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54163 (local address 127.0.97.2:43977)
0924 19:00:03.765726 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:03.765732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:03.766745 (+ 1013us) server_negotiation.cc:407] Connection header received
0924 19:00:03.766878 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:03.766883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:03.766957 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:03.767040 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:03.767614 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.768392 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.770604 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.771787 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.771906 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:03.772007 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:03.772374 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 19:00:03.772496 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 19:00:03.772981 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:03.765716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43977 (local address 127.0.97.1:54163)
0924 19:00:03.766580 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:03.766615 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:03.766753 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:03.767079 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:03.767083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:03.767103 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:03.767432 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:03.767440 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.768522 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.768527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:03.770441 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:03.770454 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.771976 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.771981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:03.772198 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:03.772256 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 19:00:03.772337 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 19:00:03.772506 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":118,"thread_start_us":645,"threads_started":1}
I20260924 19:00:03.773002 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:03.765590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42637 (local address 127.0.97.1:50895)
0924 19:00:03.766513 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:03.766533 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:03.766690 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:03.767233 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:03.767236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:03.767257 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:03.767664 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:03.767678 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.768685 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.768690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:03.770292 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:03.770303 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.771889 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.771897 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:03.772100 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:03.772238 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 19:00:03.772337 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 19:00:03.772506 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":79,"thread_start_us":319,"threads_started":1}
I20260924 19:00:03.773973 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d79252fac4450985859064776b1513" candidate_uuid: "f819faf5f5fa4b339163667ef2d94ff4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9691585dd5b94d62aeab6b7bc1929fae" is_pre_election: true
I20260924 19:00:03.773977 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d79252fac4450985859064776b1513" candidate_uuid: "f819faf5f5fa4b339163667ef2d94ff4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" is_pre_election: true
I20260924 19:00:03.774502 2581 raft_consensus.cc:2454] T 95d79252fac4450985859064776b1513 P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f819faf5f5fa4b339163667ef2d94ff4 in term 0.
I20260924 19:00:03.774520 2442 raft_consensus.cc:2454] T 95d79252fac4450985859064776b1513 P f622b4e172b6419eb1ae8033dcbd8c41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f819faf5f5fa4b339163667ef2d94ff4 in term 0.
I20260924 19:00:03.775177 2235 leader_election.cc:302] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9691585dd5b94d62aeab6b7bc1929fae, f819faf5f5fa4b339163667ef2d94ff4; no voters:
I20260924 19:00:03.775523 2657 raft_consensus.cc:2781] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:00:03.775669 2657 raft_consensus.cc:482] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:00:03.775781 2657 raft_consensus.cc:3037] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:03.777928 2657 raft_consensus.cc:504] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:03.778439 2657 leader_election.cc:288] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4 [CANDIDATE]: Term 1 election: Requested vote from peers f622b4e172b6419eb1ae8033dcbd8c41 (127.0.97.2:43977), 9691585dd5b94d62aeab6b7bc1929fae (127.0.97.3:42637)
I20260924 19:00:03.778854 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d79252fac4450985859064776b1513" candidate_uuid: "f819faf5f5fa4b339163667ef2d94ff4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f622b4e172b6419eb1ae8033dcbd8c41"
I20260924 19:00:03.778872 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d79252fac4450985859064776b1513" candidate_uuid: "f819faf5f5fa4b339163667ef2d94ff4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9691585dd5b94d62aeab6b7bc1929fae"
I20260924 19:00:03.779043 2442 raft_consensus.cc:3037] T 95d79252fac4450985859064776b1513 P f622b4e172b6419eb1ae8033dcbd8c41 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:03.779052 2581 raft_consensus.cc:3037] T 95d79252fac4450985859064776b1513 P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:03.781147 2581 raft_consensus.cc:2454] T 95d79252fac4450985859064776b1513 P 9691585dd5b94d62aeab6b7bc1929fae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f819faf5f5fa4b339163667ef2d94ff4 in term 1.
I20260924 19:00:03.781478 2442 raft_consensus.cc:2454] T 95d79252fac4450985859064776b1513 P f622b4e172b6419eb1ae8033dcbd8c41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f819faf5f5fa4b339163667ef2d94ff4 in term 1.
I20260924 19:00:03.781541 2235 leader_election.cc:302] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9691585dd5b94d62aeab6b7bc1929fae, f819faf5f5fa4b339163667ef2d94ff4; no voters:
I20260924 19:00:03.781840 2657 raft_consensus.cc:2781] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:00:03.782307 2657 raft_consensus.cc:686] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4 [term 1 LEADER]: Becoming Leader. State: Replica: f819faf5f5fa4b339163667ef2d94ff4, State: Running, Role: LEADER
I20260924 19:00:03.782811 2657 consensus_queue.cc:237] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:03.787181 2146 catalog_manager.cc:5495] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f819faf5f5fa4b339163667ef2d94ff4 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "f819faf5f5fa4b339163667ef2d94ff4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } health_report { overall_health: UNKNOWN } } }
Sep 24 19:00:03 dist-test-slave-9fn1 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276401, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:03.828485 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:03.815744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33983 (local address 127.0.0.1:39748)
0924 19:00:03.816345 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:03.816359 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:03.816511 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:03.816808 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:03.816812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:03.817476 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:03.817780 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:03.817795 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.818751 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.818759 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:03.819813 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:03.819825 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.820565 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.820572 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:03.820733 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:03.821482 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:03.821543 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:03.823558 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:03.826456 (+ 2898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:03.826465 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:03.826471 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:03.826880 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:03.827180 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:03.827184 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:03.827187 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:03.827302 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:03.827734 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:03.827739 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:03.827869 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 19:00:03.827995 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 19:00:03.828177 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":359,"threads_started":1}
I20260924 19:00:03.828563 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:03.815758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39748 (local address 127.0.97.62:33983)
0924 19:00:03.815929 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:03.815934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:03.816440 (+ 506us) server_negotiation.cc:407] Connection header received
0924 19:00:03.816630 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:03.816639 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:03.816696 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:03.816795 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:03.817917 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.818622 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.819942 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:03.820345 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:03.820485 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:03.823720 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:03.823725 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:03.823728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:03.823755 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:03.826332 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:03.826976 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:03.826979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:03.826981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:03.827091 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:03.827387 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:03.827389 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:03.827391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:03.827627 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:03.827729 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:03.828080 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 19:00:03.828297 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":65}
I20260924 19:00:03.833262 2146 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39748:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 19:00:03.834178 2146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:00:03.840059 2283 tablet_service.cc:1467] Processing CreateTablet for tablet a6c0e0922f0e4349ab535ee155b7b36d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d61ec79dc7584e8a9b7a8a9dd82380d0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:00:03.840096 2422 tablet_service.cc:1467] Processing CreateTablet for tablet a6c0e0922f0e4349ab535ee155b7b36d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d61ec79dc7584e8a9b7a8a9dd82380d0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:00:03.840330 2561 tablet_service.cc:1467] Processing CreateTablet for tablet a6c0e0922f0e4349ab535ee155b7b36d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d61ec79dc7584e8a9b7a8a9dd82380d0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:00:03.840921 2283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6c0e0922f0e4349ab535ee155b7b36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:03.840996 2422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6c0e0922f0e4349ab535ee155b7b36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:03.841023 2561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6c0e0922f0e4349ab535ee155b7b36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:03.846098 2653 tablet_bootstrap.cc:492] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4: Bootstrap starting.
I20260924 19:00:03.847672 2653 tablet_bootstrap.cc:654] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:03.847776 2651 tablet_bootstrap.cc:492] T a6c0e0922f0e4349ab535ee155b7b36d P 9691585dd5b94d62aeab6b7bc1929fae: Bootstrap starting.
I20260924 19:00:03.849159 2650 tablet_bootstrap.cc:492] T a6c0e0922f0e4349ab535ee155b7b36d P f622b4e172b6419eb1ae8033dcbd8c41: Bootstrap starting.
I20260924 19:00:03.849315 2651 tablet_bootstrap.cc:654] T a6c0e0922f0e4349ab535ee155b7b36d P 9691585dd5b94d62aeab6b7bc1929fae: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:03.851035 2650 tablet_bootstrap.cc:654] T a6c0e0922f0e4349ab535ee155b7b36d P f622b4e172b6419eb1ae8033dcbd8c41: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:03.851235 2653 tablet_bootstrap.cc:492] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4: No bootstrap required, opened a new log
I20260924 19:00:03.851394 2653 ts_tablet_manager.cc:1397] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260924 19:00:03.852279 2651 tablet_bootstrap.cc:492] T a6c0e0922f0e4349ab535ee155b7b36d P 9691585dd5b94d62aeab6b7bc1929fae: No bootstrap required, opened a new log
I20260924 19:00:03.852219 2653 raft_consensus.cc:348] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:03.852456 2651 ts_tablet_manager.cc:1397] T a6c0e0922f0e4349ab535ee155b7b36d P 9691585dd5b94d62aeab6b7bc1929fae: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260924 19:00:03.852473 2653 raft_consensus.cc:374] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:03.852578 2653 raft_consensus.cc:729] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f819faf5f5fa4b339163667ef2d94ff4, State: Initialized, Role: FOLLOWER
I20260924 19:00:03.852810 2653 consensus_queue.cc:260] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:03.853113 2651 raft_consensus.cc:348] T a6c0e0922f0e4349ab535ee155b7b36d P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:03.853325 2651 raft_consensus.cc:374] T a6c0e0922f0e4349ab535ee155b7b36d P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:03.853440 2651 raft_consensus.cc:729] T a6c0e0922f0e4349ab535ee155b7b36d P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9691585dd5b94d62aeab6b7bc1929fae, State: Initialized, Role: FOLLOWER
I20260924 19:00:03.853688 2651 consensus_queue.cc:260] T a6c0e0922f0e4349ab535ee155b7b36d P 9691585dd5b94d62aeab6b7bc1929fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:03.853823 2657 tablet_replica.cc:439] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } } }
I20260924 19:00:03.854144 2657 tablet_replica.cc:442] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:00:03.854305 2653 ts_tablet_manager.cc:1428] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 19:00:03.854156 2656 tablet_replica.cc:439] T a6c0e0922f0e4349ab535ee155b7b36d P 9691585dd5b94d62aeab6b7bc1929fae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } } }
I20260924 19:00:03.854457 2656 tablet_replica.cc:442] T a6c0e0922f0e4349ab535ee155b7b36d P 9691585dd5b94d62aeab6b7bc1929fae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:00:03.855329 2651 ts_tablet_manager.cc:1428] T a6c0e0922f0e4349ab535ee155b7b36d P 9691585dd5b94d62aeab6b7bc1929fae: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 19:00:03.856246 2650 tablet_bootstrap.cc:492] T a6c0e0922f0e4349ab535ee155b7b36d P f622b4e172b6419eb1ae8033dcbd8c41: No bootstrap required, opened a new log
I20260924 19:00:03.856401 2650 ts_tablet_manager.cc:1397] T a6c0e0922f0e4349ab535ee155b7b36d P f622b4e172b6419eb1ae8033dcbd8c41: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 19:00:03.857091 2650 raft_consensus.cc:348] T a6c0e0922f0e4349ab535ee155b7b36d P f622b4e172b6419eb1ae8033dcbd8c41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:03.857326 2650 raft_consensus.cc:374] T a6c0e0922f0e4349ab535ee155b7b36d P f622b4e172b6419eb1ae8033dcbd8c41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:03.857434 2650 raft_consensus.cc:729] T a6c0e0922f0e4349ab535ee155b7b36d P f622b4e172b6419eb1ae8033dcbd8c41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f622b4e172b6419eb1ae8033dcbd8c41, State: Initialized, Role: FOLLOWER
I20260924 19:00:03.857858 2650 consensus_queue.cc:260] T a6c0e0922f0e4349ab535ee155b7b36d P f622b4e172b6419eb1ae8033dcbd8c41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:03.858539 2650 ts_tablet_manager.cc:1428] T a6c0e0922f0e4349ab535ee155b7b36d P f622b4e172b6419eb1ae8033dcbd8c41: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:00:03.858421 2658 tablet_replica.cc:439] T a6c0e0922f0e4349ab535ee155b7b36d P f622b4e172b6419eb1ae8033dcbd8c41: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } } }
I20260924 19:00:03.858824 2658 tablet_replica.cc:442] T a6c0e0922f0e4349ab535ee155b7b36d P f622b4e172b6419eb1ae8033dcbd8c41: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:00:04.114578 2656 raft_consensus.cc:482] T a6c0e0922f0e4349ab535ee155b7b36d P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:00:04.114768 2656 raft_consensus.cc:504] T a6c0e0922f0e4349ab535ee155b7b36d P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:04.116336 2656 leader_election.cc:288] T a6c0e0922f0e4349ab535ee155b7b36d P 9691585dd5b94d62aeab6b7bc1929fae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f819faf5f5fa4b339163667ef2d94ff4 (127.0.97.1:32905), f622b4e172b6419eb1ae8033dcbd8c41 (127.0.97.2:43977)
I20260924 19:00:04.122453 2657 raft_consensus.cc:482] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:00:04.122654 2657 raft_consensus.cc:504] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:04.123294 2657 leader_election.cc:288] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f622b4e172b6419eb1ae8033dcbd8c41 (127.0.97.2:43977), 9691585dd5b94d62aeab6b7bc1929fae (127.0.97.3:42637)
I20260924 19:00:04.123353 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:04.116581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32905 (local address 127.0.97.3:39875)
0924 19:00:04.117046 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:04.117068 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:04.117193 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:04.117690 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:04.117695 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:04.117729 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:04.117985 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:04.117997 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:04.119584 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:04.119587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:04.121288 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:04.121300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:04.122484 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:04.122492 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:04.122719 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:04.122753 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 19:00:04.122860 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 19:00:04.122990 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":55,"thread_start_us":255,"threads_started":1}
I20260924 19:00:04.123340 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:04.116725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39875 (local address 127.0.97.1:32905)
0924 19:00:04.117263 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:04.117270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:04.117294 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:00:04.117386 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:04.117392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:04.117483 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:04.118151 (+ 668us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:04.118288 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:04.119469 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:04.121440 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:04.122265 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:04.122661 (+ 396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:04.122757 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:04.122882 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 19:00:04.123010 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":258,"threads_started":1}
I20260924 19:00:04.123716 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c0e0922f0e4349ab535ee155b7b36d" candidate_uuid: "f819faf5f5fa4b339163667ef2d94ff4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" is_pre_election: true
I20260924 19:00:04.123922 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c0e0922f0e4349ab535ee155b7b36d" candidate_uuid: "f819faf5f5fa4b339163667ef2d94ff4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9691585dd5b94d62aeab6b7bc1929fae" is_pre_election: true
I20260924 19:00:04.123982 2442 raft_consensus.cc:2454] T a6c0e0922f0e4349ab535ee155b7b36d P f622b4e172b6419eb1ae8033dcbd8c41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f819faf5f5fa4b339163667ef2d94ff4 in term 0.
I20260924 19:00:04.124159 2581 raft_consensus.cc:2454] T a6c0e0922f0e4349ab535ee155b7b36d P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f819faf5f5fa4b339163667ef2d94ff4 in term 0.
I20260924 19:00:04.124388 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:04.117233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43977 (local address 127.0.97.3:60751)
0924 19:00:04.117744 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:04.117757 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:04.117856 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:04.118152 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:04.118155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:04.118171 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:04.118391 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:04.118398 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:04.119766 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:04.119770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:04.121365 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:04.121376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:04.122524 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:04.122529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:04.123655 (+ 1126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:04.123688 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 19:00:04.123803 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 19:00:04.123992 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":22,"thread_start_us":278,"threads_started":1}
I20260924 19:00:04.124410 2236 leader_election.cc:302] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f622b4e172b6419eb1ae8033dcbd8c41, f819faf5f5fa4b339163667ef2d94ff4; no voters:
I20260924 19:00:04.124711 2657 raft_consensus.cc:2781] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:00:04.124809 2657 raft_consensus.cc:482] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:00:04.124917 2657 raft_consensus.cc:3037] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:04.124984 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:04.117493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60751 (local address 127.0.97.2:43977)
0924 19:00:04.117709 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:04.117715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:04.117838 (+ 123us) server_negotiation.cc:407] Connection header received
0924 19:00:04.117954 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:04.117959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:04.118041 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:04.118138 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:04.118533 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:04.119579 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:04.121527 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:04.122315 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:04.122473 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:04.122573 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:04.124421 (+ 1848us) server_negotiation.cc:299] Negotiation successful
0924 19:00:04.124712 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 19:00:04.125305 2303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c0e0922f0e4349ab535ee155b7b36d" candidate_uuid: "9691585dd5b94d62aeab6b7bc1929fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f819faf5f5fa4b339163667ef2d94ff4" is_pre_election: true
I20260924 19:00:04.125942 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c0e0922f0e4349ab535ee155b7b36d" candidate_uuid: "9691585dd5b94d62aeab6b7bc1929fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" is_pre_election: true
I20260924 19:00:04.126168 2442 raft_consensus.cc:2454] T a6c0e0922f0e4349ab535ee155b7b36d P f622b4e172b6419eb1ae8033dcbd8c41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9691585dd5b94d62aeab6b7bc1929fae in term 0.
I20260924 19:00:04.126598 2515 leader_election.cc:302] T a6c0e0922f0e4349ab535ee155b7b36d P 9691585dd5b94d62aeab6b7bc1929fae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9691585dd5b94d62aeab6b7bc1929fae, f622b4e172b6419eb1ae8033dcbd8c41; no voters:
I20260924 19:00:04.126998 2656 raft_consensus.cc:2781] T a6c0e0922f0e4349ab535ee155b7b36d P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:00:04.127122 2656 raft_consensus.cc:482] T a6c0e0922f0e4349ab535ee155b7b36d P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:00:04.127202 2656 raft_consensus.cc:3037] T a6c0e0922f0e4349ab535ee155b7b36d P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:04.127908 2657 raft_consensus.cc:504] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:04.128351 2303 raft_consensus.cc:2379] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9691585dd5b94d62aeab6b7bc1929fae in current term 1: Already voted for candidate f819faf5f5fa4b339163667ef2d94ff4 in this term.
I20260924 19:00:04.128535 2657 leader_election.cc:288] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4 [CANDIDATE]: Term 1 election: Requested vote from peers f622b4e172b6419eb1ae8033dcbd8c41 (127.0.97.2:43977), 9691585dd5b94d62aeab6b7bc1929fae (127.0.97.3:42637)
I20260924 19:00:04.128988 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c0e0922f0e4349ab535ee155b7b36d" candidate_uuid: "f819faf5f5fa4b339163667ef2d94ff4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f622b4e172b6419eb1ae8033dcbd8c41"
I20260924 19:00:04.129187 2442 raft_consensus.cc:3037] T a6c0e0922f0e4349ab535ee155b7b36d P f622b4e172b6419eb1ae8033dcbd8c41 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:04.129791 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c0e0922f0e4349ab535ee155b7b36d" candidate_uuid: "f819faf5f5fa4b339163667ef2d94ff4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9691585dd5b94d62aeab6b7bc1929fae"
I20260924 19:00:04.130103 2656 raft_consensus.cc:504] T a6c0e0922f0e4349ab535ee155b7b36d P 9691585dd5b94d62aeab6b7bc1929fae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:04.130510 2581 raft_consensus.cc:2379] T a6c0e0922f0e4349ab535ee155b7b36d P 9691585dd5b94d62aeab6b7bc1929fae [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f819faf5f5fa4b339163667ef2d94ff4 in current term 1: Already voted for candidate 9691585dd5b94d62aeab6b7bc1929fae in this term.
I20260924 19:00:04.130718 2656 leader_election.cc:288] T a6c0e0922f0e4349ab535ee155b7b36d P 9691585dd5b94d62aeab6b7bc1929fae [CANDIDATE]: Term 1 election: Requested vote from peers f819faf5f5fa4b339163667ef2d94ff4 (127.0.97.1:32905), f622b4e172b6419eb1ae8033dcbd8c41 (127.0.97.2:43977)
I20260924 19:00:04.131546 2441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c0e0922f0e4349ab535ee155b7b36d" candidate_uuid: "9691585dd5b94d62aeab6b7bc1929fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f622b4e172b6419eb1ae8033dcbd8c41"
I20260924 19:00:04.131847 2442 raft_consensus.cc:2454] T a6c0e0922f0e4349ab535ee155b7b36d P f622b4e172b6419eb1ae8033dcbd8c41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f819faf5f5fa4b339163667ef2d94ff4 in term 1.
I20260924 19:00:04.132221 2236 leader_election.cc:302] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f622b4e172b6419eb1ae8033dcbd8c41, f819faf5f5fa4b339163667ef2d94ff4; no voters: 9691585dd5b94d62aeab6b7bc1929fae
I20260924 19:00:04.132284 2303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6c0e0922f0e4349ab535ee155b7b36d" candidate_uuid: "9691585dd5b94d62aeab6b7bc1929fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f819faf5f5fa4b339163667ef2d94ff4"
I20260924 19:00:04.132500 2657 raft_consensus.cc:2781] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:00:04.132661 2657 raft_consensus.cc:686] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4 [term 1 LEADER]: Becoming Leader. State: Replica: f819faf5f5fa4b339163667ef2d94ff4, State: Running, Role: LEADER
I20260924 19:00:04.132930 2657 consensus_queue.cc:237] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:04.134521 2515 leader_election.cc:302] T a6c0e0922f0e4349ab535ee155b7b36d P 9691585dd5b94d62aeab6b7bc1929fae [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9691585dd5b94d62aeab6b7bc1929fae; no voters: f622b4e172b6419eb1ae8033dcbd8c41, f819faf5f5fa4b339163667ef2d94ff4
I20260924 19:00:04.134956 2656 raft_consensus.cc:2726] T a6c0e0922f0e4349ab535ee155b7b36d P 9691585dd5b94d62aeab6b7bc1929fae [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 19:00:04.134730 2661 tablet_replica.cc:439] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4: TxnStatusTablet state changed. Reason: New leader f819faf5f5fa4b339163667ef2d94ff4. Latest consensus state: current_term: 1 leader_uuid: "f819faf5f5fa4b339163667ef2d94ff4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } } }
I20260924 19:00:04.135070 2661 tablet_replica.cc:442] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:00:04.135730 2676 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 19:00:04.135998 2676 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 19:00:04.136075 2145 catalog_manager.cc:5495] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f819faf5f5fa4b339163667ef2d94ff4 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "f819faf5f5fa4b339163667ef2d94ff4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } health_report { overall_health: UNKNOWN } } }
I20260924 19:00:04.222505 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:04.216762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32905 (local address 127.0.0.1:36496)
0924 19:00:04.217007 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:04.217027 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:04.217181 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:04.217696 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:04.217700 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:04.217711 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:00:04.217897 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:04.217906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:04.218976 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:04.218984 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:04.220099 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:04.220110 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:04.220826 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:04.220832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:04.220988 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:04.221096 (+ 108us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:00:04.221993 (+ 897us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:00:04.222012 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 19:00:04.222085 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 19:00:04.222225 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 19:00:04.222506 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:04.216833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36496 (local address 127.0.97.1:32905)
0924 19:00:04.217031 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:04.217037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:04.217148 (+ 111us) server_negotiation.cc:407] Connection header received
0924 19:00:04.217394 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:04.217400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:04.217486 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:04.217680 (+ 194us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:00:04.218024 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:04.218839 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:04.220231 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:04.220666 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:04.220812 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:04.221291 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:00:04.221886 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:00:04.222056 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:04.222117 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 19:00:04.222192 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 19:00:04.229959 2581 raft_consensus.cc:1261] T a6c0e0922f0e4349ab535ee155b7b36d P 9691585dd5b94d62aeab6b7bc1929fae [term 1 FOLLOWER]: Refusing update from remote peer f819faf5f5fa4b339163667ef2d94ff4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:00:04.230021 2441 raft_consensus.cc:1261] T a6c0e0922f0e4349ab535ee155b7b36d P f622b4e172b6419eb1ae8033dcbd8c41 [term 1 FOLLOWER]: Refusing update from remote peer f819faf5f5fa4b339163667ef2d94ff4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:00:04.230732 2661 consensus_queue.cc:1035] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:00:04.231066 2657 consensus_queue.cc:1035] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:00:04.241632 2658 tablet_replica.cc:439] T a6c0e0922f0e4349ab535ee155b7b36d P f622b4e172b6419eb1ae8033dcbd8c41: TxnStatusTablet state changed. Reason: New leader f819faf5f5fa4b339163667ef2d94ff4. Latest consensus state: current_term: 1 leader_uuid: "f819faf5f5fa4b339163667ef2d94ff4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } } }
I20260924 19:00:04.241923 2658 tablet_replica.cc:442] T a6c0e0922f0e4349ab535ee155b7b36d P f622b4e172b6419eb1ae8033dcbd8c41: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:00:04.243681 2656 tablet_replica.cc:439] T a6c0e0922f0e4349ab535ee155b7b36d P 9691585dd5b94d62aeab6b7bc1929fae: TxnStatusTablet state changed. Reason: New leader f819faf5f5fa4b339163667ef2d94ff4. Latest consensus state: current_term: 1 leader_uuid: "f819faf5f5fa4b339163667ef2d94ff4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } } }
I20260924 19:00:04.243981 2656 tablet_replica.cc:442] T a6c0e0922f0e4349ab535ee155b7b36d P 9691585dd5b94d62aeab6b7bc1929fae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:00:04.245102 2679 mvcc.cc:205] Tried to move back new op lower bound from 7332972151717175296 to 7332972151330484224. Current Snapshot: MvccSnapshot[applied={T|T < 7332972151717175296}]
I20260924 19:00:04.246507 2681 mvcc.cc:205] Tried to move back new op lower bound from 7332972151717175296 to 7332972151330484224. Current Snapshot: MvccSnapshot[applied={T|T < 7332972151717175296}]
I20260924 19:00:04.254855 2658 tablet_replica.cc:439] T a6c0e0922f0e4349ab535ee155b7b36d P f622b4e172b6419eb1ae8033dcbd8c41: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f819faf5f5fa4b339163667ef2d94ff4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } } }
I20260924 19:00:04.255100 2658 tablet_replica.cc:442] T a6c0e0922f0e4349ab535ee155b7b36d P f622b4e172b6419eb1ae8033dcbd8c41: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:00:04.257633 2661 tablet_replica.cc:439] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f819faf5f5fa4b339163667ef2d94ff4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } } }
I20260924 19:00:04.257898 2661 tablet_replica.cc:442] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:00:04.258508 2657 tablet_replica.cc:439] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f819faf5f5fa4b339163667ef2d94ff4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } } }
I20260924 19:00:04.258836 2657 tablet_replica.cc:442] T a6c0e0922f0e4349ab535ee155b7b36d P f819faf5f5fa4b339163667ef2d94ff4: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:00:04.262921 2656 tablet_replica.cc:439] T a6c0e0922f0e4349ab535ee155b7b36d P 9691585dd5b94d62aeab6b7bc1929fae: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f819faf5f5fa4b339163667ef2d94ff4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } } }
I20260924 19:00:04.263172 2656 tablet_replica.cc:442] T a6c0e0922f0e4349ab535ee155b7b36d P 9691585dd5b94d62aeab6b7bc1929fae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:00:04.272614 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:04.266560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36504 (local address 127.0.97.1:32905)
0924 19:00:04.266778 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:04.266788 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:04.267248 (+ 460us) server_negotiation.cc:407] Connection header received
0924 19:00:04.267440 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:04.267443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:04.267525 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:04.267607 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:00:04.268045 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:04.268847 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:04.270869 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:04.271269 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:04.271438 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:04.271844 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:00:04.271945 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:00:04.272038 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:04.272229 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 19:00:04.272316 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 19:00:04.279392 2657 consensus_queue.cc:1035] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:00:04.285411 2661 consensus_queue.cc:1035] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:00:04.295933 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:04.284501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38275 (local address 127.0.97.1:32905)
0924 19:00:04.287525 (+ 3024us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:04.287531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:04.290137 (+ 2606us) server_negotiation.cc:407] Connection header received
0924 19:00:04.290334 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:04.290339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:04.290414 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:04.290501 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:00:04.291152 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:04.291918 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:04.294143 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:04.294638 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:04.294730 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:04.295271 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:00:04.295359 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:00:04.295425 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:04.295627 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 19:00:04.295687 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2903}
I20260924 19:00:04.296277 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:04.283904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32905 (local address 127.0.97.1:38275)
0924 19:00:04.290026 (+ 6122us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:04.290046 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:04.290186 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:04.290636 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:04.290641 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:04.290653 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:00:04.290961 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:04.290973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:04.292064 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:04.292071 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:04.294003 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:04.294017 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:04.294794 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:04.294802 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:04.294982 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:04.295053 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:00:04.295479 (+ 426us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:00:04.295490 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 19:00:04.295577 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 19:00:04.295716 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5939,"thread_start_us":5803,"threads_started":1}
I20260924 19:00:04.354815 2145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:00:04.355082 2145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:00:04.358635 2145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P edba9d3537dc4d52b49c6117e3562a89: Sending DeleteTablet for 3 replicas of tablet 95d79252fac4450985859064776b1513
I20260924 19:00:04.359817 2422 tablet_service.cc:1514] Processing DeleteTablet for tablet 95d79252fac4450985859064776b1513 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:04 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43378
I20260924 19:00:04.360337 2283 tablet_service.cc:1514] Processing DeleteTablet for tablet 95d79252fac4450985859064776b1513 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:04 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36486
I20260924 19:00:04.360846 2561 tablet_service.cc:1514] Processing DeleteTablet for tablet 95d79252fac4450985859064776b1513 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:04 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41822
I20260924 19:00:04.361167 2144 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:00:04.361811 2144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:00:04.366231 2699 tablet_replica.cc:330] T 95d79252fac4450985859064776b1513 P 9691585dd5b94d62aeab6b7bc1929fae: stopping tablet replica
I20260924 19:00:04.367300 2699 raft_consensus.cc:2229] T 95d79252fac4450985859064776b1513 P 9691585dd5b94d62aeab6b7bc1929fae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:00:04.368399 2699 raft_consensus.cc:2258] T 95d79252fac4450985859064776b1513 P 9691585dd5b94d62aeab6b7bc1929fae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:04.369562 2561 tablet_service.cc:1467] Processing CreateTablet for tablet accf0335dc9f47f0ad9d1a3c529e953a (DEFAULT_TABLE table=test-table [id=7b7c0aa44e994a2caa785b6ed0382b6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:04.370038 2561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet accf0335dc9f47f0ad9d1a3c529e953a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:04.371333 2421 tablet_service.cc:1467] Processing CreateTablet for tablet accf0335dc9f47f0ad9d1a3c529e953a (DEFAULT_TABLE table=test-table [id=7b7c0aa44e994a2caa785b6ed0382b6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:04.371364 2282 tablet_service.cc:1467] Processing CreateTablet for tablet accf0335dc9f47f0ad9d1a3c529e953a (DEFAULT_TABLE table=test-table [id=7b7c0aa44e994a2caa785b6ed0382b6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:04.371716 2421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet accf0335dc9f47f0ad9d1a3c529e953a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:04.371850 2282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet accf0335dc9f47f0ad9d1a3c529e953a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:04.382009 2697 tablet_replica.cc:330] T 95d79252fac4450985859064776b1513 P f622b4e172b6419eb1ae8033dcbd8c41: stopping tablet replica
I20260924 19:00:04.382522 2697 raft_consensus.cc:2229] T 95d79252fac4450985859064776b1513 P f622b4e172b6419eb1ae8033dcbd8c41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:00:04.382954 2697 raft_consensus.cc:2258] T 95d79252fac4450985859064776b1513 P f622b4e172b6419eb1ae8033dcbd8c41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:04.383808 2700 tablet_bootstrap.cc:492] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae: Bootstrap starting.
I20260924 19:00:04.384141 2702 tablet_bootstrap.cc:492] T accf0335dc9f47f0ad9d1a3c529e953a P f819faf5f5fa4b339163667ef2d94ff4: Bootstrap starting.
I20260924 19:00:04.386415 2701 tablet_bootstrap.cc:492] T accf0335dc9f47f0ad9d1a3c529e953a P f622b4e172b6419eb1ae8033dcbd8c41: Bootstrap starting.
I20260924 19:00:04.387233 2699 ts_tablet_manager.cc:1905] T 95d79252fac4450985859064776b1513 P 9691585dd5b94d62aeab6b7bc1929fae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:04.387274 2702 tablet_bootstrap.cc:654] T accf0335dc9f47f0ad9d1a3c529e953a P f819faf5f5fa4b339163667ef2d94ff4: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:04.389102 2701 tablet_bootstrap.cc:654] T accf0335dc9f47f0ad9d1a3c529e953a P f622b4e172b6419eb1ae8033dcbd8c41: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:04.388104 2700 tablet_bootstrap.cc:654] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:04.389638 2702 tablet_bootstrap.cc:492] T accf0335dc9f47f0ad9d1a3c529e953a P f819faf5f5fa4b339163667ef2d94ff4: No bootstrap required, opened a new log
I20260924 19:00:04.389770 2702 ts_tablet_manager.cc:1397] T accf0335dc9f47f0ad9d1a3c529e953a P f819faf5f5fa4b339163667ef2d94ff4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 19:00:04.391587 2702 raft_consensus.cc:348] T accf0335dc9f47f0ad9d1a3c529e953a P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:04.391858 2702 raft_consensus.cc:374] T accf0335dc9f47f0ad9d1a3c529e953a P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:04.391959 2702 raft_consensus.cc:729] T accf0335dc9f47f0ad9d1a3c529e953a P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f819faf5f5fa4b339163667ef2d94ff4, State: Initialized, Role: FOLLOWER
I20260924 19:00:04.392216 2702 consensus_queue.cc:260] T accf0335dc9f47f0ad9d1a3c529e953a P f819faf5f5fa4b339163667ef2d94ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:04.396026 2702 ts_tablet_manager.cc:1428] T accf0335dc9f47f0ad9d1a3c529e953a P f819faf5f5fa4b339163667ef2d94ff4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260924 19:00:04.387233 2697 ts_tablet_manager.cc:1905] T 95d79252fac4450985859064776b1513 P f622b4e172b6419eb1ae8033dcbd8c41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:04.399641 2700 tablet_bootstrap.cc:492] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae: No bootstrap required, opened a new log
I20260924 19:00:04.399811 2700 ts_tablet_manager.cc:1397] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20260924 19:00:04.400524 2700 raft_consensus.cc:348] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:04.400755 2700 raft_consensus.cc:374] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:04.400856 2700 raft_consensus.cc:729] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9691585dd5b94d62aeab6b7bc1929fae, State: Initialized, Role: FOLLOWER
I20260924 19:00:04.401096 2700 consensus_queue.cc:260] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:04.401683 2700 ts_tablet_manager.cc:1428] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:00:04.401746 2701 tablet_bootstrap.cc:492] T accf0335dc9f47f0ad9d1a3c529e953a P f622b4e172b6419eb1ae8033dcbd8c41: No bootstrap required, opened a new log
I20260924 19:00:04.401882 2701 ts_tablet_manager.cc:1397] T accf0335dc9f47f0ad9d1a3c529e953a P f622b4e172b6419eb1ae8033dcbd8c41: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.011s
I20260924 19:00:04.402387 2698 tablet_replica.cc:330] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4: stopping tablet replica
I20260924 19:00:04.402842 2698 raft_consensus.cc:2229] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:00:04.403498 2698 raft_consensus.cc:2258] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:04.404486 2701 raft_consensus.cc:348] T accf0335dc9f47f0ad9d1a3c529e953a P f622b4e172b6419eb1ae8033dcbd8c41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:04.404722 2701 raft_consensus.cc:374] T accf0335dc9f47f0ad9d1a3c529e953a P f622b4e172b6419eb1ae8033dcbd8c41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:04.404824 2701 raft_consensus.cc:729] T accf0335dc9f47f0ad9d1a3c529e953a P f622b4e172b6419eb1ae8033dcbd8c41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f622b4e172b6419eb1ae8033dcbd8c41, State: Initialized, Role: FOLLOWER
I20260924 19:00:04.406124 2697 ts_tablet_manager.cc:1918] T 95d79252fac4450985859064776b1513 P f622b4e172b6419eb1ae8033dcbd8c41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:00:04.406275 2697 log.cc:1196] T 95d79252fac4450985859064776b1513 P f622b4e172b6419eb1ae8033dcbd8c41: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-1/wal/wals/95d79252fac4450985859064776b1513
I20260924 19:00:04.406683 2698 ts_tablet_manager.cc:1905] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:04.406776 2697 ts_tablet_manager.cc:1939] T 95d79252fac4450985859064776b1513 P f622b4e172b6419eb1ae8033dcbd8c41: Deleting consensus metadata
I20260924 19:00:04.408443 2133 catalog_manager.cc:4841] TS f622b4e172b6419eb1ae8033dcbd8c41 (127.0.97.2:43977): tablet 95d79252fac4450985859064776b1513 (table test-table [id=9fed71d60b7f41df992d1f7757cc08a7]) successfully deleted
I20260924 19:00:04.408514 2701 consensus_queue.cc:260] T accf0335dc9f47f0ad9d1a3c529e953a P f622b4e172b6419eb1ae8033dcbd8c41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:04.410450 2701 ts_tablet_manager.cc:1428] T accf0335dc9f47f0ad9d1a3c529e953a P f622b4e172b6419eb1ae8033dcbd8c41: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260924 19:00:04.414211 2698 ts_tablet_manager.cc:1918] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:00:04.414384 2698 log.cc:1196] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-0/wal/wals/95d79252fac4450985859064776b1513
I20260924 19:00:04.414917 2698 ts_tablet_manager.cc:1939] T 95d79252fac4450985859064776b1513 P f819faf5f5fa4b339163667ef2d94ff4: Deleting consensus metadata
I20260924 19:00:04.415092 2699 ts_tablet_manager.cc:1918] T 95d79252fac4450985859064776b1513 P 9691585dd5b94d62aeab6b7bc1929fae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:00:04.415274 2699 log.cc:1196] T 95d79252fac4450985859064776b1513 P 9691585dd5b94d62aeab6b7bc1929fae: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-2/wal/wals/95d79252fac4450985859064776b1513
I20260924 19:00:04.415766 2699 ts_tablet_manager.cc:1939] T 95d79252fac4450985859064776b1513 P 9691585dd5b94d62aeab6b7bc1929fae: Deleting consensus metadata
I20260924 19:00:04.416510 2133 catalog_manager.cc:4841] TS f819faf5f5fa4b339163667ef2d94ff4 (127.0.97.1:32905): tablet 95d79252fac4450985859064776b1513 (table test-table [id=9fed71d60b7f41df992d1f7757cc08a7]) successfully deleted
I20260924 19:00:04.417374 2132 catalog_manager.cc:4841] TS 9691585dd5b94d62aeab6b7bc1929fae (127.0.97.3:42637): tablet 95d79252fac4450985859064776b1513 (table test-table [id=9fed71d60b7f41df992d1f7757cc08a7]) successfully deleted
I20260924 19:00:04.568934 2656 raft_consensus.cc:482] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:00:04.569168 2656 raft_consensus.cc:504] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:04.569823 2656 leader_election.cc:288] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f819faf5f5fa4b339163667ef2d94ff4 (127.0.97.1:32905), f622b4e172b6419eb1ae8033dcbd8c41 (127.0.97.2:43977)
I20260924 19:00:04.570402 2303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "accf0335dc9f47f0ad9d1a3c529e953a" candidate_uuid: "9691585dd5b94d62aeab6b7bc1929fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f819faf5f5fa4b339163667ef2d94ff4" is_pre_election: true
I20260924 19:00:04.570518 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "accf0335dc9f47f0ad9d1a3c529e953a" candidate_uuid: "9691585dd5b94d62aeab6b7bc1929fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" is_pre_election: true
I20260924 19:00:04.570675 2303 raft_consensus.cc:2454] T accf0335dc9f47f0ad9d1a3c529e953a P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9691585dd5b94d62aeab6b7bc1929fae in term 0.
I20260924 19:00:04.570742 2442 raft_consensus.cc:2454] T accf0335dc9f47f0ad9d1a3c529e953a P f622b4e172b6419eb1ae8033dcbd8c41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9691585dd5b94d62aeab6b7bc1929fae in term 0.
I20260924 19:00:04.571100 2515 leader_election.cc:302] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9691585dd5b94d62aeab6b7bc1929fae, f819faf5f5fa4b339163667ef2d94ff4; no voters:
I20260924 19:00:04.571388 2656 raft_consensus.cc:2781] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:00:04.571492 2656 raft_consensus.cc:482] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:00:04.571597 2656 raft_consensus.cc:3037] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:04.573695 2656 raft_consensus.cc:504] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:04.574298 2656 leader_election.cc:288] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae [CANDIDATE]: Term 1 election: Requested vote from peers f819faf5f5fa4b339163667ef2d94ff4 (127.0.97.1:32905), f622b4e172b6419eb1ae8033dcbd8c41 (127.0.97.2:43977)
I20260924 19:00:04.574800 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "accf0335dc9f47f0ad9d1a3c529e953a" candidate_uuid: "9691585dd5b94d62aeab6b7bc1929fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f622b4e172b6419eb1ae8033dcbd8c41"
I20260924 19:00:04.574782 2303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "accf0335dc9f47f0ad9d1a3c529e953a" candidate_uuid: "9691585dd5b94d62aeab6b7bc1929fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f819faf5f5fa4b339163667ef2d94ff4"
I20260924 19:00:04.575001 2442 raft_consensus.cc:3037] T accf0335dc9f47f0ad9d1a3c529e953a P f622b4e172b6419eb1ae8033dcbd8c41 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:04.575021 2303 raft_consensus.cc:3037] T accf0335dc9f47f0ad9d1a3c529e953a P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:04.576984 2442 raft_consensus.cc:2454] T accf0335dc9f47f0ad9d1a3c529e953a P f622b4e172b6419eb1ae8033dcbd8c41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9691585dd5b94d62aeab6b7bc1929fae in term 1.
I20260924 19:00:04.577350 2515 leader_election.cc:302] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9691585dd5b94d62aeab6b7bc1929fae, f622b4e172b6419eb1ae8033dcbd8c41; no voters:
I20260924 19:00:04.577629 2656 raft_consensus.cc:2781] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:00:04.578171 2656 raft_consensus.cc:686] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae [term 1 LEADER]: Becoming Leader. State: Replica: 9691585dd5b94d62aeab6b7bc1929fae, State: Running, Role: LEADER
I20260924 19:00:04.578238 2303 raft_consensus.cc:2454] T accf0335dc9f47f0ad9d1a3c529e953a P f819faf5f5fa4b339163667ef2d94ff4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9691585dd5b94d62aeab6b7bc1929fae in term 1.
I20260924 19:00:04.578534 2656 consensus_queue.cc:237] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:04.582770 2145 catalog_manager.cc:5495] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae reported cstate change: term changed from 0 to 1, leader changed from <none> to 9691585dd5b94d62aeab6b7bc1929fae (127.0.97.3). New cstate: current_term: 1 leader_uuid: "9691585dd5b94d62aeab6b7bc1929fae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } health_report { overall_health: HEALTHY } } }
I20260924 19:00:04.639982 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:04.629558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38322 (local address 127.0.97.3:42637)
0924 19:00:04.634319 (+ 4761us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:04.634325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:04.634345 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:00:04.634450 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:04.634456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:04.634578 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:04.634786 (+ 208us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:00:04.635122 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:04.636217 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:04.637493 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:04.638165 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:04.638293 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:04.638642 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:00:04.639357 (+ 715us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:00:04.639501 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:04.639563 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 19:00:04.639642 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4607,"thread_start_us":4503,"threads_started":1}
I20260924 19:00:04.656062 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:04.649859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32905 (local address 127.0.97.3:40979)
0924 19:00:04.650517 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:04.650532 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:04.650638 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:04.650983 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:04.650986 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:04.650995 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:00:04.651260 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:04.651267 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:04.652748 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:04.652752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:04.653930 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:04.653942 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:04.654982 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:04.654987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:04.655123 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:04.655177 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:00:04.655624 (+ 447us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:00:04.655632 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 19:00:04.655691 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 19:00:04.655786 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":350,"threads_started":1}
I20260924 19:00:04.656121 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:04.649887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40979 (local address 127.0.97.1:32905)
0924 19:00:04.650078 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:04.650086 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:04.650609 (+ 523us) server_negotiation.cc:407] Connection header received
0924 19:00:04.650768 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:04.650774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:04.650872 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:04.650967 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:00:04.651404 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:04.652572 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:04.654065 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:04.654807 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:04.654933 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:04.655353 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:00:04.655514 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:00:04.655610 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:04.655735 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 19:00:04.655821 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 19:00:04.664845 2442 raft_consensus.cc:1261] T accf0335dc9f47f0ad9d1a3c529e953a P f622b4e172b6419eb1ae8033dcbd8c41 [term 1 FOLLOWER]: Refusing update from remote peer 9691585dd5b94d62aeab6b7bc1929fae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:00:04.665234 2303 raft_consensus.cc:1261] T accf0335dc9f47f0ad9d1a3c529e953a P f819faf5f5fa4b339163667ef2d94ff4 [term 1 FOLLOWER]: Refusing update from remote peer 9691585dd5b94d62aeab6b7bc1929fae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:00:04.665971 2706 consensus_queue.cc:1035] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae [LEADER]: Connected to new peer: Peer: permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:00:04.666667 2656 consensus_queue.cc:1035] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae [LEADER]: Connected to new peer: Peer: permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:00:04.671151 2680 mvcc.cc:205] Tried to move back new op lower bound from 7332972153501573120 to 7332972153158303744. Current Snapshot: MvccSnapshot[applied={T|T < 7332972153501573120}]
I20260924 19:00:04.697577 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:04.691774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44209 (local address 127.0.97.3:42637)
0924 19:00:04.691944 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:04.691952 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:04.692338 (+ 386us) server_negotiation.cc:407] Connection header received
0924 19:00:04.692398 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:04.692403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:04.692476 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:04.692565 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:00:04.693099 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:04.693905 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:04.695776 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:04.696418 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:04.696553 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:04.696887 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:00:04.697006 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:00:04.697099 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:04.697214 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 19:00:04.697289 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 19:00:04.697610 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:04.691800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42637 (local address 127.0.97.1:44209)
0924 19:00:04.692105 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:04.692124 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:04.692236 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:04.692722 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:04.692728 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:04.692741 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:00:04.692927 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:04.692938 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:04.694042 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:04.694049 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:04.695618 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:04.695630 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:04.696573 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:04.696578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:04.696689 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:04.696732 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:00:04.697101 (+ 369us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:00:04.697108 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 19:00:04.697196 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 19:00:04.697328 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175}
W20260924 19:00:04.698932 2680 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 19:00:06.287902 2145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:00:06.288199 2145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:00:06.292057 2145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P edba9d3537dc4d52b49c6117e3562a89: Sending DeleteTablet for 3 replicas of tablet accf0335dc9f47f0ad9d1a3c529e953a
I20260924 19:00:06.292761 2282 tablet_service.cc:1514] Processing DeleteTablet for tablet accf0335dc9f47f0ad9d1a3c529e953a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36486
I20260924 19:00:06.292774 2561 tablet_service.cc:1514] Processing DeleteTablet for tablet accf0335dc9f47f0ad9d1a3c529e953a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41822
I20260924 19:00:06.292797 2421 tablet_service.cc:1514] Processing DeleteTablet for tablet accf0335dc9f47f0ad9d1a3c529e953a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43378
I20260924 19:00:06.305297 2741 tablet_replica.cc:330] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae: stopping tablet replica
I20260924 19:00:06.305539 2739 tablet_replica.cc:330] T accf0335dc9f47f0ad9d1a3c529e953a P f622b4e172b6419eb1ae8033dcbd8c41: stopping tablet replica
I20260924 19:00:06.305578 2741 raft_consensus.cc:2229] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:00:06.305815 2739 raft_consensus.cc:2229] T accf0335dc9f47f0ad9d1a3c529e953a P f622b4e172b6419eb1ae8033dcbd8c41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:00:06.306092 2741 raft_consensus.cc:2258] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:06.306121 2739 raft_consensus.cc:2258] T accf0335dc9f47f0ad9d1a3c529e953a P f622b4e172b6419eb1ae8033dcbd8c41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:06.306700 2740 tablet_replica.cc:330] T accf0335dc9f47f0ad9d1a3c529e953a P f819faf5f5fa4b339163667ef2d94ff4: stopping tablet replica
I20260924 19:00:06.306942 2740 raft_consensus.cc:2229] T accf0335dc9f47f0ad9d1a3c529e953a P f819faf5f5fa4b339163667ef2d94ff4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:00:06.307180 2741 ts_tablet_manager.cc:1905] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:06.307214 2740 raft_consensus.cc:2258] T accf0335dc9f47f0ad9d1a3c529e953a P f819faf5f5fa4b339163667ef2d94ff4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:06.307973 2739 ts_tablet_manager.cc:1905] T accf0335dc9f47f0ad9d1a3c529e953a P f622b4e172b6419eb1ae8033dcbd8c41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:06.308221 2740 ts_tablet_manager.cc:1905] T accf0335dc9f47f0ad9d1a3c529e953a P f819faf5f5fa4b339163667ef2d94ff4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:06.314023 2741 ts_tablet_manager.cc:1918] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:00:06.314159 2741 log.cc:1196] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-2/wal/wals/accf0335dc9f47f0ad9d1a3c529e953a
I20260924 19:00:06.314615 2741 ts_tablet_manager.cc:1939] T accf0335dc9f47f0ad9d1a3c529e953a P 9691585dd5b94d62aeab6b7bc1929fae: Deleting consensus metadata
I20260924 19:00:06.314744 2740 ts_tablet_manager.cc:1918] T accf0335dc9f47f0ad9d1a3c529e953a P f819faf5f5fa4b339163667ef2d94ff4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:00:06.314960 2740 log.cc:1196] T accf0335dc9f47f0ad9d1a3c529e953a P f819faf5f5fa4b339163667ef2d94ff4: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-0/wal/wals/accf0335dc9f47f0ad9d1a3c529e953a
I20260924 19:00:06.315356 2739 ts_tablet_manager.cc:1918] T accf0335dc9f47f0ad9d1a3c529e953a P f622b4e172b6419eb1ae8033dcbd8c41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:00:06.315500 2739 log.cc:1196] T accf0335dc9f47f0ad9d1a3c529e953a P f622b4e172b6419eb1ae8033dcbd8c41: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-1/wal/wals/accf0335dc9f47f0ad9d1a3c529e953a
I20260924 19:00:06.315495 2740 ts_tablet_manager.cc:1939] T accf0335dc9f47f0ad9d1a3c529e953a P f819faf5f5fa4b339163667ef2d94ff4: Deleting consensus metadata
I20260924 19:00:06.315822 2132 catalog_manager.cc:4841] TS 9691585dd5b94d62aeab6b7bc1929fae (127.0.97.3:42637): tablet accf0335dc9f47f0ad9d1a3c529e953a (table test-table [id=7b7c0aa44e994a2caa785b6ed0382b6c]) successfully deleted
I20260924 19:00:06.315996 2739 ts_tablet_manager.cc:1939] T accf0335dc9f47f0ad9d1a3c529e953a P f622b4e172b6419eb1ae8033dcbd8c41: Deleting consensus metadata
I20260924 19:00:06.316653 2133 catalog_manager.cc:4841] TS f819faf5f5fa4b339163667ef2d94ff4 (127.0.97.1:32905): tablet accf0335dc9f47f0ad9d1a3c529e953a (table test-table [id=7b7c0aa44e994a2caa785b6ed0382b6c]) successfully deleted
I20260924 19:00:06.317142 2133 catalog_manager.cc:4841] TS f622b4e172b6419eb1ae8033dcbd8c41 (127.0.97.2:43977): tablet accf0335dc9f47f0ad9d1a3c529e953a (table test-table [id=7b7c0aa44e994a2caa785b6ed0382b6c]) successfully deleted
Sep 24 19:00:06 dist-test-slave-9fn1 krb5kdc[2101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276406, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 19:00:06 dist-test-slave-9fn1 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276406, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:06.351308 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:06.336015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33674 (local address 127.0.97.62:33983)
0924 19:00:06.336542 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:06.336546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:06.336568 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:00:06.336704 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:06.336707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:06.336766 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:06.336861 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:06.338217 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:06.338964 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:06.340107 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:06.340470 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:06.340563 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:06.344714 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:06.344719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:06.344722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:06.344753 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:06.347879 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:06.348665 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:06.348671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:06.348674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:06.348804 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:06.349188 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:06.349193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:06.349196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:06.349536 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:06.349758 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:06.350762 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0924 19:00:06.350964 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":289,"threads_started":1}
I20260924 19:00:06.355562 2145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:00:06.356158 2145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:00:06.362293 2282 tablet_service.cc:1467] Processing CreateTablet for tablet 77caa627e49644f58114e54fe59789d7 (DEFAULT_TABLE table=test-table [id=f6294363a5a249ceb61c9fa07e9d6076]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:06.362882 2282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77caa627e49644f58114e54fe59789d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:06.362877 2421 tablet_service.cc:1467] Processing CreateTablet for tablet 77caa627e49644f58114e54fe59789d7 (DEFAULT_TABLE table=test-table [id=f6294363a5a249ceb61c9fa07e9d6076]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:06.362949 2561 tablet_service.cc:1467] Processing CreateTablet for tablet 77caa627e49644f58114e54fe59789d7 (DEFAULT_TABLE table=test-table [id=f6294363a5a249ceb61c9fa07e9d6076]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:06.363281 2421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77caa627e49644f58114e54fe59789d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:06.363339 2561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77caa627e49644f58114e54fe59789d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:06.367924 2750 tablet_bootstrap.cc:492] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41: Bootstrap starting.
I20260924 19:00:06.369656 2750 tablet_bootstrap.cc:654] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:06.370265 2751 tablet_bootstrap.cc:492] T 77caa627e49644f58114e54fe59789d7 P 9691585dd5b94d62aeab6b7bc1929fae: Bootstrap starting.
I20260924 19:00:06.371807 2752 tablet_bootstrap.cc:492] T 77caa627e49644f58114e54fe59789d7 P f819faf5f5fa4b339163667ef2d94ff4: Bootstrap starting.
I20260924 19:00:06.372171 2751 tablet_bootstrap.cc:654] T 77caa627e49644f58114e54fe59789d7 P 9691585dd5b94d62aeab6b7bc1929fae: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:06.373611 2752 tablet_bootstrap.cc:654] T 77caa627e49644f58114e54fe59789d7 P f819faf5f5fa4b339163667ef2d94ff4: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:06.376477 2750 tablet_bootstrap.cc:492] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41: No bootstrap required, opened a new log
I20260924 19:00:06.376632 2750 ts_tablet_manager.cc:1397] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260924 19:00:06.377280 2750 raft_consensus.cc:348] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:06.377543 2750 raft_consensus.cc:374] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:06.377640 2750 raft_consensus.cc:729] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f622b4e172b6419eb1ae8033dcbd8c41, State: Initialized, Role: FOLLOWER
I20260924 19:00:06.377859 2750 consensus_queue.cc:260] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:06.378640 2750 ts_tablet_manager.cc:1428] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:00:06.379104 2752 tablet_bootstrap.cc:492] T 77caa627e49644f58114e54fe59789d7 P f819faf5f5fa4b339163667ef2d94ff4: No bootstrap required, opened a new log
I20260924 19:00:06.379261 2752 ts_tablet_manager.cc:1397] T 77caa627e49644f58114e54fe59789d7 P f819faf5f5fa4b339163667ef2d94ff4: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260924 19:00:06.379972 2751 tablet_bootstrap.cc:492] T 77caa627e49644f58114e54fe59789d7 P 9691585dd5b94d62aeab6b7bc1929fae: No bootstrap required, opened a new log
I20260924 19:00:06.380119 2751 ts_tablet_manager.cc:1397] T 77caa627e49644f58114e54fe59789d7 P 9691585dd5b94d62aeab6b7bc1929fae: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260924 19:00:06.380004 2752 raft_consensus.cc:348] T 77caa627e49644f58114e54fe59789d7 P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:06.380259 2752 raft_consensus.cc:374] T 77caa627e49644f58114e54fe59789d7 P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:06.380358 2752 raft_consensus.cc:729] T 77caa627e49644f58114e54fe59789d7 P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f819faf5f5fa4b339163667ef2d94ff4, State: Initialized, Role: FOLLOWER
I20260924 19:00:06.380580 2752 consensus_queue.cc:260] T 77caa627e49644f58114e54fe59789d7 P f819faf5f5fa4b339163667ef2d94ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:06.380888 2751 raft_consensus.cc:348] T 77caa627e49644f58114e54fe59789d7 P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:06.381101 2751 raft_consensus.cc:374] T 77caa627e49644f58114e54fe59789d7 P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:06.381135 2752 ts_tablet_manager.cc:1428] T 77caa627e49644f58114e54fe59789d7 P f819faf5f5fa4b339163667ef2d94ff4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 19:00:06.381192 2751 raft_consensus.cc:729] T 77caa627e49644f58114e54fe59789d7 P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9691585dd5b94d62aeab6b7bc1929fae, State: Initialized, Role: FOLLOWER
I20260924 19:00:06.381423 2751 consensus_queue.cc:260] T 77caa627e49644f58114e54fe59789d7 P 9691585dd5b94d62aeab6b7bc1929fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:06.381978 2751 ts_tablet_manager.cc:1428] T 77caa627e49644f58114e54fe59789d7 P 9691585dd5b94d62aeab6b7bc1929fae: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:00:06.384174 2756 raft_consensus.cc:482] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:00:06.384375 2756 raft_consensus.cc:504] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:06.386114 2756 leader_election.cc:288] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f819faf5f5fa4b339163667ef2d94ff4 (127.0.97.1:32905), 9691585dd5b94d62aeab6b7bc1929fae (127.0.97.3:42637)
I20260924 19:00:06.396612 2759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:06.386606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45189 (local address 127.0.97.1:32905)
0924 19:00:06.390514 (+ 3908us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:06.390523 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:06.390552 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:00:06.390632 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:06.390640 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:06.390734 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:06.390867 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:06.391388 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:06.392635 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:06.394748 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:06.395567 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:06.395708 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:06.395841 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:06.396138 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 19:00:06.396265 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3754,"thread_start_us":3591,"threads_started":1}
I20260924 19:00:06.396665 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:06.386241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32905 (local address 127.0.97.2:45189)
0924 19:00:06.386814 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:06.386836 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:06.386975 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:06.390884 (+ 3909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:06.390888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:06.390913 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:06.391186 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:06.391194 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:06.392783 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:06.392788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:06.394594 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:06.394606 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:06.395757 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:06.395762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:06.395960 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:06.395995 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 19:00:06.396103 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 19:00:06.396237 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":5,"thread_start_us":323,"threads_started":1}
I20260924 19:00:06.397367 2303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77caa627e49644f58114e54fe59789d7" candidate_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f819faf5f5fa4b339163667ef2d94ff4" is_pre_election: true
I20260924 19:00:06.397562 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:06.386748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41285 (local address 127.0.97.3:42637)
0924 19:00:06.390000 (+ 3252us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:06.390008 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:06.391669 (+ 1661us) server_negotiation.cc:407] Connection header received
0924 19:00:06.391893 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:06.391898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:06.391977 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:06.392099 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:06.392609 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:06.393581 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:06.395439 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:06.396313 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:06.396456 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:06.396561 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:06.397120 (+ 559us) server_negotiation.cc:299] Negotiation successful
0924 19:00:06.397257 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3088,"thread_start_us":2955,"threads_started":1}
I20260924 19:00:06.397619 2758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:06.386373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42637 (local address 127.0.97.2:41285)
0924 19:00:06.391550 (+ 5177us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:06.391565 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:06.391742 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:06.392114 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:06.392118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:06.392143 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:06.392447 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:06.392455 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:06.393823 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:06.393826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:06.395247 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:06.395259 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:06.396730 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:06.396738 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:06.396945 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:06.396975 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 19:00:06.397064 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 19:00:06.397221 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5082,"mutex_wait_us":6,"thread_start_us":4891,"threads_started":1}
I20260924 19:00:06.397712 2303 raft_consensus.cc:2454] T 77caa627e49644f58114e54fe59789d7 P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f622b4e172b6419eb1ae8033dcbd8c41 in term 0.
I20260924 19:00:06.398217 2376 leader_election.cc:302] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f622b4e172b6419eb1ae8033dcbd8c41, f819faf5f5fa4b339163667ef2d94ff4; no voters:
I20260924 19:00:06.398289 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77caa627e49644f58114e54fe59789d7" candidate_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9691585dd5b94d62aeab6b7bc1929fae" is_pre_election: true
I20260924 19:00:06.398494 2581 raft_consensus.cc:2454] T 77caa627e49644f58114e54fe59789d7 P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f622b4e172b6419eb1ae8033dcbd8c41 in term 0.
I20260924 19:00:06.398649 2756 raft_consensus.cc:2781] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:00:06.398762 2756 raft_consensus.cc:482] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:00:06.398892 2756 raft_consensus.cc:3037] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:06.401461 2756 raft_consensus.cc:504] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:06.402105 2756 leader_election.cc:288] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41 [CANDIDATE]: Term 1 election: Requested vote from peers f819faf5f5fa4b339163667ef2d94ff4 (127.0.97.1:32905), 9691585dd5b94d62aeab6b7bc1929fae (127.0.97.3:42637)
I20260924 19:00:06.402510 2303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77caa627e49644f58114e54fe59789d7" candidate_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f819faf5f5fa4b339163667ef2d94ff4"
I20260924 19:00:06.402527 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77caa627e49644f58114e54fe59789d7" candidate_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9691585dd5b94d62aeab6b7bc1929fae"
I20260924 19:00:06.402716 2303 raft_consensus.cc:3037] T 77caa627e49644f58114e54fe59789d7 P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:06.402746 2581 raft_consensus.cc:3037] T 77caa627e49644f58114e54fe59789d7 P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:06.404935 2303 raft_consensus.cc:2454] T 77caa627e49644f58114e54fe59789d7 P f819faf5f5fa4b339163667ef2d94ff4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f622b4e172b6419eb1ae8033dcbd8c41 in term 1.
I20260924 19:00:06.405153 2581 raft_consensus.cc:2454] T 77caa627e49644f58114e54fe59789d7 P 9691585dd5b94d62aeab6b7bc1929fae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f622b4e172b6419eb1ae8033dcbd8c41 in term 1.
I20260924 19:00:06.405306 2376 leader_election.cc:302] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f622b4e172b6419eb1ae8033dcbd8c41, f819faf5f5fa4b339163667ef2d94ff4; no voters:
I20260924 19:00:06.405614 2756 raft_consensus.cc:2781] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:00:06.406044 2756 raft_consensus.cc:686] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41 [term 1 LEADER]: Becoming Leader. State: Replica: f622b4e172b6419eb1ae8033dcbd8c41, State: Running, Role: LEADER
I20260924 19:00:06.406421 2756 consensus_queue.cc:237] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:06.409951 2145 catalog_manager.cc:5495] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41 reported cstate change: term changed from 0 to 1, leader changed from <none> to f622b4e172b6419eb1ae8033dcbd8c41 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } health_report { overall_health: UNKNOWN } } }
I20260924 19:00:06.438014 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:06.432056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40198 (local address 127.0.97.2:43977)
0924 19:00:06.432580 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:06.432584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:06.432611 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:00:06.432688 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:06.432691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:06.432758 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:06.432898 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:00:06.433394 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:06.434358 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:06.435722 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:06.436132 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:06.436219 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:06.436644 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:00:06.437249 (+ 605us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:00:06.437353 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:06.437584 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 19:00:06.437652 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":283,"threads_started":1}
I20260924 19:00:06.453015 2759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:06.447900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48349 (local address 127.0.97.1:32905)
0924 19:00:06.448136 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:06.448144 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:06.448455 (+ 311us) server_negotiation.cc:407] Connection header received
0924 19:00:06.448668 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:06.448672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:06.448732 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:06.448824 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:00:06.449254 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:06.450065 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:06.451394 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:06.451783 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:06.451911 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:06.452321 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:00:06.452431 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:00:06.452524 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:06.452653 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 19:00:06.452724 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 19:00:06.453133 2767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:06.447665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:32905 (local address 127.0.97.2:48349)
0924 19:00:06.448361 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:06.448376 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:06.448528 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:06.448836 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:06.448840 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:06.448852 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:00:06.449123 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:06.449130 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:06.450198 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:06.450203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:06.451259 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:06.451269 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:06.451958 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:06.451967 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:06.452104 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:06.452150 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:00:06.452525 (+ 375us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:00:06.452537 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 19:00:06.452634 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 19:00:06.452757 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":422,"threads_started":1}
I20260924 19:00:06.460812 2303 raft_consensus.cc:1261] T 77caa627e49644f58114e54fe59789d7 P f819faf5f5fa4b339163667ef2d94ff4 [term 1 FOLLOWER]: Refusing update from remote peer f622b4e172b6419eb1ae8033dcbd8c41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:00:06.461391 2581 raft_consensus.cc:1261] T 77caa627e49644f58114e54fe59789d7 P 9691585dd5b94d62aeab6b7bc1929fae [term 1 FOLLOWER]: Refusing update from remote peer f622b4e172b6419eb1ae8033dcbd8c41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:00:06.461467 2761 consensus_queue.cc:1035] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:00:06.461946 2756 consensus_queue.cc:1035] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:00:06.491340 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:06.481251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57909 (local address 127.0.97.2:43977)
0924 19:00:06.481651 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:06.481656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:06.485453 (+ 3797us) server_negotiation.cc:407] Connection header received
0924 19:00:06.485628 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:06.485632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:06.485687 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:06.485757 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:00:06.486306 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:06.487183 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:06.489121 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:06.489735 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:06.489883 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:06.490694 (+ 811us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:00:06.490807 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:00:06.490860 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:06.491063 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 19:00:06.491123 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294}
I20260924 19:00:06.491552 2772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:06.481210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43977 (local address 127.0.97.1:57909)
0924 19:00:06.485345 (+ 4135us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:06.485364 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:06.485524 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:06.485806 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:06.485811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:06.485824 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:00:06.486150 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:06.486160 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:06.487293 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:06.487297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:06.488923 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:06.488937 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:06.490276 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:06.490283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:06.490478 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:06.490523 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:00:06.490929 (+ 406us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:00:06.490938 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 19:00:06.491007 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 19:00:06.491146 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3968,"thread_start_us":3863,"threads_started":1}
I20260924 19:00:06.523607 2144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:00:06.523847 2144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:00:06.527720 2144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P edba9d3537dc4d52b49c6117e3562a89: Sending DeleteTablet for 3 replicas of tablet 77caa627e49644f58114e54fe59789d7
I20260924 19:00:06.528702 2282 tablet_service.cc:1514] Processing DeleteTablet for tablet 77caa627e49644f58114e54fe59789d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36486
I20260924 19:00:06.528775 2561 tablet_service.cc:1514] Processing DeleteTablet for tablet 77caa627e49644f58114e54fe59789d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41822
I20260924 19:00:06.529008 2740 tablet_replica.cc:330] T 77caa627e49644f58114e54fe59789d7 P f819faf5f5fa4b339163667ef2d94ff4: stopping tablet replica
I20260924 19:00:06.529039 2741 tablet_replica.cc:330] T 77caa627e49644f58114e54fe59789d7 P 9691585dd5b94d62aeab6b7bc1929fae: stopping tablet replica
I20260924 19:00:06.529206 2741 raft_consensus.cc:2229] T 77caa627e49644f58114e54fe59789d7 P 9691585dd5b94d62aeab6b7bc1929fae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:00:06.529206 2740 raft_consensus.cc:2229] T 77caa627e49644f58114e54fe59789d7 P f819faf5f5fa4b339163667ef2d94ff4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:00:06.529374 2740 raft_consensus.cc:2258] T 77caa627e49644f58114e54fe59789d7 P f819faf5f5fa4b339163667ef2d94ff4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:06.529374 2741 raft_consensus.cc:2258] T 77caa627e49644f58114e54fe59789d7 P 9691585dd5b94d62aeab6b7bc1929fae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:06.530100 2421 tablet_service.cc:1514] Processing DeleteTablet for tablet 77caa627e49644f58114e54fe59789d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43378
I20260924 19:00:06.530326 2741 ts_tablet_manager.cc:1905] T 77caa627e49644f58114e54fe59789d7 P 9691585dd5b94d62aeab6b7bc1929fae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:06.530567 2740 ts_tablet_manager.cc:1905] T 77caa627e49644f58114e54fe59789d7 P f819faf5f5fa4b339163667ef2d94ff4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:06.535676 2741 ts_tablet_manager.cc:1918] T 77caa627e49644f58114e54fe59789d7 P 9691585dd5b94d62aeab6b7bc1929fae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:00:06.535811 2741 log.cc:1196] T 77caa627e49644f58114e54fe59789d7 P 9691585dd5b94d62aeab6b7bc1929fae: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-2/wal/wals/77caa627e49644f58114e54fe59789d7
I20260924 19:00:06.536178 2741 ts_tablet_manager.cc:1939] T 77caa627e49644f58114e54fe59789d7 P 9691585dd5b94d62aeab6b7bc1929fae: Deleting consensus metadata
I20260924 19:00:06.537213 2739 tablet_replica.cc:330] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41: stopping tablet replica
I20260924 19:00:06.537415 2739 raft_consensus.cc:2229] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:00:06.538609 2739 raft_consensus.cc:2258] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:06.539739 2739 ts_tablet_manager.cc:1905] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:06.545735 2740 ts_tablet_manager.cc:1918] T 77caa627e49644f58114e54fe59789d7 P f819faf5f5fa4b339163667ef2d94ff4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:00:06.545702 2132 catalog_manager.cc:4841] TS 9691585dd5b94d62aeab6b7bc1929fae (127.0.97.3:42637): tablet 77caa627e49644f58114e54fe59789d7 (table test-table [id=f6294363a5a249ceb61c9fa07e9d6076]) successfully deleted
I20260924 19:00:06.545961 2740 log.cc:1196] T 77caa627e49644f58114e54fe59789d7 P f819faf5f5fa4b339163667ef2d94ff4: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-0/wal/wals/77caa627e49644f58114e54fe59789d7
I20260924 19:00:06.546419 2740 ts_tablet_manager.cc:1939] T 77caa627e49644f58114e54fe59789d7 P f819faf5f5fa4b339163667ef2d94ff4: Deleting consensus metadata
I20260924 19:00:06.546375 2144 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:00:06.547094 2144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:00:06.547712 2133 catalog_manager.cc:4841] TS f819faf5f5fa4b339163667ef2d94ff4 (127.0.97.1:32905): tablet 77caa627e49644f58114e54fe59789d7 (table test-table [id=f6294363a5a249ceb61c9fa07e9d6076]) successfully deleted
I20260924 19:00:06.549381 2739 ts_tablet_manager.cc:1918] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:00:06.549623 2739 log.cc:1196] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-1/wal/wals/77caa627e49644f58114e54fe59789d7
I20260924 19:00:06.550096 2739 ts_tablet_manager.cc:1939] T 77caa627e49644f58114e54fe59789d7 P f622b4e172b6419eb1ae8033dcbd8c41: Deleting consensus metadata
I20260924 19:00:06.551195 2133 catalog_manager.cc:4841] TS f622b4e172b6419eb1ae8033dcbd8c41 (127.0.97.2:43977): tablet 77caa627e49644f58114e54fe59789d7 (table test-table [id=f6294363a5a249ceb61c9fa07e9d6076]) successfully deleted
I20260924 19:00:06.555138 2282 tablet_service.cc:1467] Processing CreateTablet for tablet 1794b87be74e465d9ed6db46519501ac (DEFAULT_TABLE table=test-table [id=563588f065004bfebdb225fb4127a956]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:06.555297 2421 tablet_service.cc:1467] Processing CreateTablet for tablet 1794b87be74e465d9ed6db46519501ac (DEFAULT_TABLE table=test-table [id=563588f065004bfebdb225fb4127a956]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:06.555538 2282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1794b87be74e465d9ed6db46519501ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:06.555511 2561 tablet_service.cc:1467] Processing CreateTablet for tablet 1794b87be74e465d9ed6db46519501ac (DEFAULT_TABLE table=test-table [id=563588f065004bfebdb225fb4127a956]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:06.555646 2421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1794b87be74e465d9ed6db46519501ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:06.555943 2561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1794b87be74e465d9ed6db46519501ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:06.561854 2752 tablet_bootstrap.cc:492] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4: Bootstrap starting.
I20260924 19:00:06.562424 2750 tablet_bootstrap.cc:492] T 1794b87be74e465d9ed6db46519501ac P f622b4e172b6419eb1ae8033dcbd8c41: Bootstrap starting.
I20260924 19:00:06.562958 2751 tablet_bootstrap.cc:492] T 1794b87be74e465d9ed6db46519501ac P 9691585dd5b94d62aeab6b7bc1929fae: Bootstrap starting.
I20260924 19:00:06.563405 2752 tablet_bootstrap.cc:654] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:06.564162 2750 tablet_bootstrap.cc:654] T 1794b87be74e465d9ed6db46519501ac P f622b4e172b6419eb1ae8033dcbd8c41: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:06.565095 2751 tablet_bootstrap.cc:654] T 1794b87be74e465d9ed6db46519501ac P 9691585dd5b94d62aeab6b7bc1929fae: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:06.577858 2752 tablet_bootstrap.cc:492] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4: No bootstrap required, opened a new log
I20260924 19:00:06.577858 2751 tablet_bootstrap.cc:492] T 1794b87be74e465d9ed6db46519501ac P 9691585dd5b94d62aeab6b7bc1929fae: No bootstrap required, opened a new log
I20260924 19:00:06.578038 2752 ts_tablet_manager.cc:1397] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.014s
I20260924 19:00:06.578038 2751 ts_tablet_manager.cc:1397] T 1794b87be74e465d9ed6db46519501ac P 9691585dd5b94d62aeab6b7bc1929fae: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.003s
I20260924 19:00:06.578807 2752 raft_consensus.cc:348] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:06.578804 2751 raft_consensus.cc:348] T 1794b87be74e465d9ed6db46519501ac P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:06.579062 2752 raft_consensus.cc:374] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:06.579145 2752 raft_consensus.cc:729] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f819faf5f5fa4b339163667ef2d94ff4, State: Initialized, Role: FOLLOWER
I20260924 19:00:06.579432 2752 consensus_queue.cc:260] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:06.580868 2752 ts_tablet_manager.cc:1428] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 19:00:06.581338 2751 raft_consensus.cc:374] T 1794b87be74e465d9ed6db46519501ac P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:06.581519 2751 raft_consensus.cc:729] T 1794b87be74e465d9ed6db46519501ac P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9691585dd5b94d62aeab6b7bc1929fae, State: Initialized, Role: FOLLOWER
I20260924 19:00:06.581931 2751 consensus_queue.cc:260] T 1794b87be74e465d9ed6db46519501ac P 9691585dd5b94d62aeab6b7bc1929fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:06.582929 2751 ts_tablet_manager.cc:1428] T 1794b87be74e465d9ed6db46519501ac P 9691585dd5b94d62aeab6b7bc1929fae: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260924 19:00:06.585678 2750 tablet_bootstrap.cc:492] T 1794b87be74e465d9ed6db46519501ac P f622b4e172b6419eb1ae8033dcbd8c41: No bootstrap required, opened a new log
I20260924 19:00:06.585919 2750 ts_tablet_manager.cc:1397] T 1794b87be74e465d9ed6db46519501ac P f622b4e172b6419eb1ae8033dcbd8c41: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.014s
I20260924 19:00:06.586884 2750 raft_consensus.cc:348] T 1794b87be74e465d9ed6db46519501ac P f622b4e172b6419eb1ae8033dcbd8c41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:06.587159 2750 raft_consensus.cc:374] T 1794b87be74e465d9ed6db46519501ac P f622b4e172b6419eb1ae8033dcbd8c41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:06.587250 2750 raft_consensus.cc:729] T 1794b87be74e465d9ed6db46519501ac P f622b4e172b6419eb1ae8033dcbd8c41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f622b4e172b6419eb1ae8033dcbd8c41, State: Initialized, Role: FOLLOWER
I20260924 19:00:06.587560 2750 consensus_queue.cc:260] T 1794b87be74e465d9ed6db46519501ac P f622b4e172b6419eb1ae8033dcbd8c41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:06.588860 2750 ts_tablet_manager.cc:1428] T 1794b87be74e465d9ed6db46519501ac P f622b4e172b6419eb1ae8033dcbd8c41: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 19:00:06.752161 2721 raft_consensus.cc:482] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:00:06.752424 2721 raft_consensus.cc:504] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:06.753355 2721 leader_election.cc:288] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f622b4e172b6419eb1ae8033dcbd8c41 (127.0.97.2:43977), 9691585dd5b94d62aeab6b7bc1929fae (127.0.97.3:42637)
I20260924 19:00:06.753968 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1794b87be74e465d9ed6db46519501ac" candidate_uuid: "f819faf5f5fa4b339163667ef2d94ff4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9691585dd5b94d62aeab6b7bc1929fae" is_pre_election: true
I20260924 19:00:06.754015 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1794b87be74e465d9ed6db46519501ac" candidate_uuid: "f819faf5f5fa4b339163667ef2d94ff4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" is_pre_election: true
I20260924 19:00:06.754213 2581 raft_consensus.cc:2454] T 1794b87be74e465d9ed6db46519501ac P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f819faf5f5fa4b339163667ef2d94ff4 in term 0.
I20260924 19:00:06.754276 2442 raft_consensus.cc:2454] T 1794b87be74e465d9ed6db46519501ac P f622b4e172b6419eb1ae8033dcbd8c41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f819faf5f5fa4b339163667ef2d94ff4 in term 0.
I20260924 19:00:06.754702 2235 leader_election.cc:302] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9691585dd5b94d62aeab6b7bc1929fae, f819faf5f5fa4b339163667ef2d94ff4; no voters:
I20260924 19:00:06.754983 2721 raft_consensus.cc:2781] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:00:06.755553 2721 raft_consensus.cc:482] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:00:06.755658 2721 raft_consensus.cc:3037] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:06.758688 2721 raft_consensus.cc:504] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:06.759346 2721 leader_election.cc:288] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4 [CANDIDATE]: Term 1 election: Requested vote from peers f622b4e172b6419eb1ae8033dcbd8c41 (127.0.97.2:43977), 9691585dd5b94d62aeab6b7bc1929fae (127.0.97.3:42637)
I20260924 19:00:06.759882 2442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1794b87be74e465d9ed6db46519501ac" candidate_uuid: "f819faf5f5fa4b339163667ef2d94ff4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f622b4e172b6419eb1ae8033dcbd8c41"
I20260924 19:00:06.759931 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1794b87be74e465d9ed6db46519501ac" candidate_uuid: "f819faf5f5fa4b339163667ef2d94ff4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9691585dd5b94d62aeab6b7bc1929fae"
I20260924 19:00:06.760128 2442 raft_consensus.cc:3037] T 1794b87be74e465d9ed6db46519501ac P f622b4e172b6419eb1ae8033dcbd8c41 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:06.760154 2581 raft_consensus.cc:3037] T 1794b87be74e465d9ed6db46519501ac P 9691585dd5b94d62aeab6b7bc1929fae [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:06.762578 2581 raft_consensus.cc:2454] T 1794b87be74e465d9ed6db46519501ac P 9691585dd5b94d62aeab6b7bc1929fae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f819faf5f5fa4b339163667ef2d94ff4 in term 1.
I20260924 19:00:06.762993 2235 leader_election.cc:302] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9691585dd5b94d62aeab6b7bc1929fae, f819faf5f5fa4b339163667ef2d94ff4; no voters:
I20260924 19:00:06.763283 2721 raft_consensus.cc:2781] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:00:06.763303 2442 raft_consensus.cc:2454] T 1794b87be74e465d9ed6db46519501ac P f622b4e172b6419eb1ae8033dcbd8c41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f819faf5f5fa4b339163667ef2d94ff4 in term 1.
I20260924 19:00:06.763464 2721 raft_consensus.cc:686] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4 [term 1 LEADER]: Becoming Leader. State: Replica: f819faf5f5fa4b339163667ef2d94ff4, State: Running, Role: LEADER
I20260924 19:00:06.763732 2721 consensus_queue.cc:237] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } }
I20260924 19:00:06.766789 2144 catalog_manager.cc:5495] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f819faf5f5fa4b339163667ef2d94ff4 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "f819faf5f5fa4b339163667ef2d94ff4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f819faf5f5fa4b339163667ef2d94ff4" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 32905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 } health_report { overall_health: UNKNOWN } } }
I20260924 19:00:06.825229 2759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:06.820404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53164 (local address 127.0.97.1:32905)
0924 19:00:06.820892 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:06.820896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:06.820909 (+ 13us) server_negotiation.cc:407] Connection header received
0924 19:00:06.820953 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:06.820956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:06.821010 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:06.821108 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:00:06.821471 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:06.822263 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:06.823537 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:06.823958 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:06.824094 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:06.824545 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:00:06.824637 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:00:06.824732 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:06.824876 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 19:00:06.824955 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388}
I20260924 19:00:06.834436 2441 raft_consensus.cc:1261] T 1794b87be74e465d9ed6db46519501ac P f622b4e172b6419eb1ae8033dcbd8c41 [term 1 FOLLOWER]: Refusing update from remote peer f819faf5f5fa4b339163667ef2d94ff4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:00:06.834532 2581 raft_consensus.cc:1261] T 1794b87be74e465d9ed6db46519501ac P 9691585dd5b94d62aeab6b7bc1929fae [term 1 FOLLOWER]: Refusing update from remote peer f819faf5f5fa4b339163667ef2d94ff4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:00:06.835067 2778 consensus_queue.cc:1035] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9691585dd5b94d62aeab6b7bc1929fae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:00:06.835405 2726 consensus_queue.cc:1035] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f622b4e172b6419eb1ae8033dcbd8c41" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:00:06.857118 2719 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 19:00:08.449632 2144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:00:08.449851 2144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:00:08.453970 2144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P edba9d3537dc4d52b49c6117e3562a89: Sending DeleteTablet for 3 replicas of tablet 1794b87be74e465d9ed6db46519501ac
I20260924 19:00:08.454701 2282 tablet_service.cc:1514] Processing DeleteTablet for tablet 1794b87be74e465d9ed6db46519501ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36486
I20260924 19:00:08.454854 2421 tablet_service.cc:1514] Processing DeleteTablet for tablet 1794b87be74e465d9ed6db46519501ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43378
I20260924 19:00:08.454869 2561 tablet_service.cc:1514] Processing DeleteTablet for tablet 1794b87be74e465d9ed6db46519501ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41822
I20260924 19:00:08.455507 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 2217
I20260924 19:00:08.463754 2802 tablet_replica.cc:330] T 1794b87be74e465d9ed6db46519501ac P f622b4e172b6419eb1ae8033dcbd8c41: stopping tablet replica
I20260924 19:00:08.464087 2802 raft_consensus.cc:2229] T 1794b87be74e465d9ed6db46519501ac P f622b4e172b6419eb1ae8033dcbd8c41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:00:08.464481 2802 raft_consensus.cc:2258] T 1794b87be74e465d9ed6db46519501ac P f622b4e172b6419eb1ae8033dcbd8c41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:08.465111 2803 tablet_replica.cc:330] T 1794b87be74e465d9ed6db46519501ac P 9691585dd5b94d62aeab6b7bc1929fae: stopping tablet replica
I20260924 19:00:08.465354 2803 raft_consensus.cc:2229] T 1794b87be74e465d9ed6db46519501ac P 9691585dd5b94d62aeab6b7bc1929fae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:00:08.465608 2801 tablet_replica.cc:330] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4: stopping tablet replica
I20260924 19:00:08.465770 2802 ts_tablet_manager.cc:1905] T 1794b87be74e465d9ed6db46519501ac P f622b4e172b6419eb1ae8033dcbd8c41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:08.465871 2801 raft_consensus.cc:2229] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:00:08.465960 2803 raft_consensus.cc:2258] T 1794b87be74e465d9ed6db46519501ac P 9691585dd5b94d62aeab6b7bc1929fae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:08.466231 2801 raft_consensus.cc:2258] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:08.467178 2801 ts_tablet_manager.cc:1905] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:08.467402 2803 ts_tablet_manager.cc:1905] T 1794b87be74e465d9ed6db46519501ac P 9691585dd5b94d62aeab6b7bc1929fae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:08.472381 2802 ts_tablet_manager.cc:1918] T 1794b87be74e465d9ed6db46519501ac P f622b4e172b6419eb1ae8033dcbd8c41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:00:08.472577 2802 log.cc:1196] T 1794b87be74e465d9ed6db46519501ac P f622b4e172b6419eb1ae8033dcbd8c41: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-1/wal/wals/1794b87be74e465d9ed6db46519501ac
I20260924 19:00:08.473001 2802 ts_tablet_manager.cc:1939] T 1794b87be74e465d9ed6db46519501ac P f622b4e172b6419eb1ae8033dcbd8c41: Deleting consensus metadata
I20260924 19:00:08.473641 2803 ts_tablet_manager.cc:1918] T 1794b87be74e465d9ed6db46519501ac P 9691585dd5b94d62aeab6b7bc1929fae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:00:08.473707 2801 ts_tablet_manager.cc:1918] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:00:08.473809 2803 log.cc:1196] T 1794b87be74e465d9ed6db46519501ac P 9691585dd5b94d62aeab6b7bc1929fae: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-2/wal/wals/1794b87be74e465d9ed6db46519501ac
I20260924 19:00:08.473819 2801 log.cc:1196] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276386554690-388-0/minicluster-data/ts-0/wal/wals/1794b87be74e465d9ed6db46519501ac
I20260924 19:00:08.474123 2133 catalog_manager.cc:4841] TS f622b4e172b6419eb1ae8033dcbd8c41 (127.0.97.2:43977): tablet 1794b87be74e465d9ed6db46519501ac (table test-table [id=563588f065004bfebdb225fb4127a956]) successfully deleted
I20260924 19:00:08.474273 2803 ts_tablet_manager.cc:1939] T 1794b87be74e465d9ed6db46519501ac P 9691585dd5b94d62aeab6b7bc1929fae: Deleting consensus metadata
I20260924 19:00:08.474308 2801 ts_tablet_manager.cc:1939] T 1794b87be74e465d9ed6db46519501ac P f819faf5f5fa4b339163667ef2d94ff4: Deleting consensus metadata
I20260924 19:00:08.475169 2133 catalog_manager.cc:4841] TS f819faf5f5fa4b339163667ef2d94ff4 (127.0.97.1:32905): tablet 1794b87be74e465d9ed6db46519501ac (table test-table [id=563588f065004bfebdb225fb4127a956]) successfully deleted
I20260924 19:00:08.475260 2132 catalog_manager.cc:4841] TS 9691585dd5b94d62aeab6b7bc1929fae (127.0.97.3:42637): tablet 1794b87be74e465d9ed6db46519501ac (table test-table [id=563588f065004bfebdb225fb4127a956]) successfully deleted
Sep 24 19:00:08 dist-test-slave-9fn1 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276406, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 19:00:08.484423 2805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:08.457744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53168 (local address 127.0.97.1:32905)
0924 19:00:08.465360 (+ 7616us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:08.465371 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:08.465400 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:00:08.465476 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:08.465480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:08.465616 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:08.465740 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:08.467548 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:08.468652 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:08.470477 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:08.471223 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:08.471331 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:08.479335 (+ 8004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:08.479349 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:08.479366 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:08.479401 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:08.482052 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:08.482634 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:08.482641 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:08.482643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:08.482773 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:08.483116 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:08.483121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:08.483124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:08.483559 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:08.483786 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:08.484003 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 19:00:08.484162 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7479,"thread_start_us":7333,"threads_started":1}
W20260924 19:00:08.485042 2348 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53168
W20260924 19:00:08.485666 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 2217: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:00:08.485886 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 2217
I20260924 19:00:08.511960 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 2357
Sep 24 19:00:08 dist-test-slave-9fn1 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276406, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 19:00:08.528417 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:08.512805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40206 (local address 127.0.97.2:43977)
0924 19:00:08.513310 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:08.513315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:08.513331 (+ 16us) server_negotiation.cc:407] Connection header received
0924 19:00:08.513384 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:08.513388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:08.513449 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:08.513612 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:08.514799 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:08.515671 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:08.517070 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:08.517470 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:08.517642 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:08.522520 (+ 4878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:08.522533 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:08.522550 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:08.522580 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:08.525620 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:08.526229 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:08.526240 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:08.526248 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:08.526393 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:08.526779 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:08.526785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:08.526788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:08.527134 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:08.527448 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:08.527732 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 19:00:08.527964 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":271,"threads_started":1}
W20260924 19:00:08.529109 2487 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40206
W20260924 19:00:08.529830 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 2357: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:00:08.529978 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 2357
I20260924 19:00:08.555721 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 2496
Sep 24 19:00:08 dist-test-slave-9fn1 krb5kdc[2101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276406, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 19:00:08.570416 2807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:08.556424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38330 (local address 127.0.97.3:42637)
0924 19:00:08.556922 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:08.556926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:08.556940 (+ 14us) server_negotiation.cc:407] Connection header received
0924 19:00:08.556990 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:08.556993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:08.557048 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:08.557166 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:08.558434 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:08.559129 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:08.560322 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:08.560756 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:08.560940 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:08.565043 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:08.565064 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:08.565079 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:08.565116 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:08.567738 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:08.568420 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:08.568431 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:08.568439 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:08.568556 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:08.568933 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:08.568939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:08.568942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:08.569281 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:08.569627 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:08.569870 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 19:00:08.570071 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":216,"threads_started":1}
W20260924 19:00:08.571058 2626 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38330
W20260924 19:00:08.571619 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 2496: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:00:08.571756 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 2496
I20260924 19:00:08.595587 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 2114
I20260924 19:00:08.596706 2186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:08.709475 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 2114
2026-09-24T19:00:08Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8510 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:00:08 dist-test-slave-9fn1 krb5kdc[2811](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:00:08 dist-test-slave-9fn1 krb5kdc[2811](info): set up 2 sockets
Sep 24 19:00:08 dist-test-slave-9fn1 krb5kdc[2811](info): commencing operation
krb5kdc: starting...
W20260924 19:00:09.805775 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:00:09 dist-test-slave-9fn1 krb5kdc[2811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:00:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:00:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 19:00:09.989428 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40969
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:10.322885 2824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:10.323145 2824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:10.323266 2824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:10.330201 2824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:00:10.330302 2824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:10.330420 2824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:10.330525 2824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:00:10.330611 2824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:00:10.340047 2824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:10.341353 2824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:10.342687 2824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:10.353734 2832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:10.353891 2829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:10.354045 2830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:10.354621 2824 server_base.cc:1034] running on GCE node
I20260924 19:00:10.355291 2824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:10.356527 2824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:10.357739 2824 hybrid_clock.cc:648] HybridClock initialized: now 1790276410357709 us; error 53 us; skew 500 ppm
I20260924 19:00:10.358054 2824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:10 dist-test-slave-9fn1 krb5kdc[2811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:10.360906 2824 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:00:10.362370 2824 webserver.cc:460] Webserver started at http://127.0.97.62:35191/ using document root <none> and password file <none>
I20260924 19:00:10.363020 2824 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:10.363158 2824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:10.363464 2824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:10.366279 2824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "8bb8ea7a71b24141883e42a2cbb29eb5"
format_stamp: "Formatted at 2026-09-24 19:00:10 on dist-test-slave-9fn1"
server_key: "40d8d955b96370b78ce85ed654807298"
server_key_iv: "177f4b0e0580c3f8b3c75cb11bc9fae6"
server_key_version: "encryptionkey@0"
I20260924 19:00:10.367038 2824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "8bb8ea7a71b24141883e42a2cbb29eb5"
format_stamp: "Formatted at 2026-09-24 19:00:10 on dist-test-slave-9fn1"
server_key: "40d8d955b96370b78ce85ed654807298"
server_key_iv: "177f4b0e0580c3f8b3c75cb11bc9fae6"
server_key_version: "encryptionkey@0"
I20260924 19:00:10.371785 2824 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 19:00:10.375038 2839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:10.376227 2824 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 19:00:10.376397 2824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "8bb8ea7a71b24141883e42a2cbb29eb5"
format_stamp: "Formatted at 2026-09-24 19:00:10 on dist-test-slave-9fn1"
server_key: "40d8d955b96370b78ce85ed654807298"
server_key_iv: "177f4b0e0580c3f8b3c75cb11bc9fae6"
server_key_version: "encryptionkey@0"
I20260924 19:00:10.376600 2824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:10.389309 2824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:10.392845 2824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:10.393121 2824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:10.410832 2824 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40969
I20260924 19:00:10.410851 2890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40969 every 8 connection(s)
I20260924 19:00:10.412168 2824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:00:10.415745 2891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:10.418743 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 2824
I20260924 19:00:10.418984 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:00:10.419777 388 external_mini_cluster.cc:1428] Setting key 6af2f37f93495a9da6c274fc7eaa58b2
I20260924 19:00:10.428315 2891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bb8ea7a71b24141883e42a2cbb29eb5: Bootstrap starting.
I20260924 19:00:10.431229 2891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bb8ea7a71b24141883e42a2cbb29eb5: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:10.432168 2891 log.cc:824] T 00000000000000000000000000000000 P 8bb8ea7a71b24141883e42a2cbb29eb5: Log is configured to *not* fsync() on all Append() calls
Sep 24 19:00:10 dist-test-slave-9fn1 krb5kdc[2811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:10.434831 2891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bb8ea7a71b24141883e42a2cbb29eb5: No bootstrap required, opened a new log
I20260924 19:00:10.442445 2891 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bb8ea7a71b24141883e42a2cbb29eb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb8ea7a71b24141883e42a2cbb29eb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40969 } }
I20260924 19:00:10.442835 2891 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bb8ea7a71b24141883e42a2cbb29eb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:10.442920 2894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:10.421905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43130 (local address 127.0.97.62:40969)
0924 19:00:10.422678 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:10.422690 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:10.422732 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:00:10.424346 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:10.424367 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:10.424801 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:10.425122 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:10.426936 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:10.428324 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:10.429859 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:10.430577 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:10.430783 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:10.435042 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:10.435050 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:10.435064 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:10.435108 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:10.438546 (+ 3438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:10.439279 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:10.439300 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:10.439307 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:10.439496 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:10.439920 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:10.439930 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:10.439932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:10.440500 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:10.440892 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:10.441356 (+ 464us) server_negotiation.cc:299] Negotiation successful
0924 19:00:10.441830 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":347,"threads_started":1}
I20260924 19:00:10.442966 2891 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bb8ea7a71b24141883e42a2cbb29eb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bb8ea7a71b24141883e42a2cbb29eb5, State: Initialized, Role: FOLLOWER
I20260924 19:00:10.443646 2891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bb8ea7a71b24141883e42a2cbb29eb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb8ea7a71b24141883e42a2cbb29eb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40969 } }
I20260924 19:00:10.443924 2891 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bb8ea7a71b24141883e42a2cbb29eb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:00:10.444058 2891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bb8ea7a71b24141883e42a2cbb29eb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:00:10.444227 2891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bb8ea7a71b24141883e42a2cbb29eb5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:10.446714 2891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bb8ea7a71b24141883e42a2cbb29eb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb8ea7a71b24141883e42a2cbb29eb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40969 } }
I20260924 19:00:10.447140 2891 leader_election.cc:302] T 00000000000000000000000000000000 P 8bb8ea7a71b24141883e42a2cbb29eb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bb8ea7a71b24141883e42a2cbb29eb5; no voters:
I20260924 19:00:10.447670 2891 leader_election.cc:288] T 00000000000000000000000000000000 P 8bb8ea7a71b24141883e42a2cbb29eb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:00:10.447793 2896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bb8ea7a71b24141883e42a2cbb29eb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:00:10.448488 2896 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bb8ea7a71b24141883e42a2cbb29eb5 [term 1 LEADER]: Becoming Leader. State: Replica: 8bb8ea7a71b24141883e42a2cbb29eb5, State: Running, Role: LEADER
I20260924 19:00:10.448993 2896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bb8ea7a71b24141883e42a2cbb29eb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb8ea7a71b24141883e42a2cbb29eb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40969 } }
I20260924 19:00:10.449734 2891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bb8ea7a71b24141883e42a2cbb29eb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:00:10.451615 2898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bb8ea7a71b24141883e42a2cbb29eb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bb8ea7a71b24141883e42a2cbb29eb5. Latest consensus state: current_term: 1 leader_uuid: "8bb8ea7a71b24141883e42a2cbb29eb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb8ea7a71b24141883e42a2cbb29eb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40969 } } }
I20260924 19:00:10.452090 2898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bb8ea7a71b24141883e42a2cbb29eb5 [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:10.452045 2897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bb8ea7a71b24141883e42a2cbb29eb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bb8ea7a71b24141883e42a2cbb29eb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bb8ea7a71b24141883e42a2cbb29eb5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40969 } } }
I20260924 19:00:10.452311 2897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bb8ea7a71b24141883e42a2cbb29eb5 [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:10.454438 2901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:00:10.460881 2901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:00:10.470575 2901 catalog_manager.cc:1347] Generated new cluster ID: 705f3ba1c297478f896cc8d7dece8f7a
I20260924 19:00:10.470709 2901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:00:10.481904 2901 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:00:10.483150 2901 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:00:10.489049 2901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bb8ea7a71b24141883e42a2cbb29eb5: Generated new TSK 0
I20260924 19:00:10.489816 2901 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 19:00:10.532985 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40969
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:10.854357 2917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:10.854647 2917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:10.854739 2917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:10.862156 2917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:10.862272 2917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:10.862387 2917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:00:10.872253 2917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:10.873472 2917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:10.874930 2917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:10.888080 2925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:10.888000 2923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:10.888162 2922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:10.889003 2917 server_base.cc:1034] running on GCE node
I20260924 19:00:10.889671 2917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:10.890605 2917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:10.891836 2917 hybrid_clock.cc:648] HybridClock initialized: now 1790276410891781 us; error 51 us; skew 500 ppm
I20260924 19:00:10.892164 2917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:10 dist-test-slave-9fn1 krb5kdc[2811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:10.895341 2917 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 19:00:10.896852 2917 webserver.cc:460] Webserver started at http://127.0.97.1:34219/ using document root <none> and password file <none>
I20260924 19:00:10.897624 2917 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:10.897769 2917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:10.898085 2917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:10.900978 2917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/data/instance:
uuid: "068b84e5ac4147e49b32e2588b4fd6cd"
format_stamp: "Formatted at 2026-09-24 19:00:10 on dist-test-slave-9fn1"
server_key: "0196578008e01f1c0ebed6e3ac60ae57"
server_key_iv: "3ce12a01b45d90640d2672a7f644e395"
server_key_version: "encryptionkey@0"
I20260924 19:00:10.901829 2917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/wal/instance:
uuid: "068b84e5ac4147e49b32e2588b4fd6cd"
format_stamp: "Formatted at 2026-09-24 19:00:10 on dist-test-slave-9fn1"
server_key: "0196578008e01f1c0ebed6e3ac60ae57"
server_key_iv: "3ce12a01b45d90640d2672a7f644e395"
server_key_version: "encryptionkey@0"
I20260924 19:00:10.907969 2917 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 19:00:10.911289 2932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:10.912520 2917 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 19:00:10.912694 2917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "068b84e5ac4147e49b32e2588b4fd6cd"
format_stamp: "Formatted at 2026-09-24 19:00:10 on dist-test-slave-9fn1"
server_key: "0196578008e01f1c0ebed6e3ac60ae57"
server_key_iv: "3ce12a01b45d90640d2672a7f644e395"
server_key_version: "encryptionkey@0"
I20260924 19:00:10.912899 2917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:10.939910 2917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:10.944188 2917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:10.944476 2917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:10.945355 2917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:10.946652 2917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:10.946754 2917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:10.946909 2917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:10.946983 2917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:10.979679 2917 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39979
I20260924 19:00:10.979784 3044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39979 every 8 connection(s)
I20260924 19:00:10.981053 2917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:00:10.990787 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 2917
I20260924 19:00:10.991020 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/wal/instance
I20260924 19:00:10.991711 388 external_mini_cluster.cc:1428] Setting key 2bbc7daa22ca35362494fcc9864a847d
Sep 24 19:00:10 dist-test-slave-9fn1 krb5kdc[2811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:11.001055 3047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:10.983528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34791 (local address 127.0.97.62:40969)
0924 19:00:10.983950 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:10.983956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:10.984935 (+ 979us) server_negotiation.cc:407] Connection header received
0924 19:00:10.986716 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:10.986720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:10.986781 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:10.986883 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:10.988283 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:10.988999 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:10.990759 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:10.991366 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:10.991472 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:10.995515 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:10.995520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:10.995523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:10.995553 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:10.998108 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:10.998842 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:10.998852 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:10.998855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:10.998969 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:10.999396 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:10.999401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:10.999404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:10.999654 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:10.999804 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:11.000491 (+ 687us) server_negotiation.cc:299] Negotiation successful
0924 19:00:11.000698 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":199,"threads_started":1}
I20260924 19:00:11.002081 3048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:10.984133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40969 (local address 127.0.97.1:34791)
0924 19:00:10.984790 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:10.984827 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:10.986485 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:10.987082 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:10.987089 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:10.987555 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:10.988089 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:10.988126 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:10.989184 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:10.989188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:10.990609 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:10.990631 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:10.991603 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:10.991611 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:10.991938 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:10.992690 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:10.992740 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:10.995300 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:10.998249 (+ 2949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:10.998258 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:10.998294 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:10.998713 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:10.999076 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:10.999083 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:10.999088 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:10.999277 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:10.999779 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:10.999787 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:11.000114 (+ 327us) client_negotiation.cc:769] Sending connection context
0924 19:00:11.000391 (+ 277us) client_negotiation.cc:240] Negotiation successful
0924 19:00:11.000915 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":268,"threads_started":1}
I20260924 19:00:11.003551 3045 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40969
I20260924 19:00:11.003875 3045 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:11.004537 3045 heartbeater.cc:511] Master 127.0.97.62:40969 requested a full tablet report, sending...
I20260924 19:00:11.006554 2856 ts_manager.cc:194] Registered new tserver with Master: 068b84e5ac4147e49b32e2588b4fd6cd (127.0.97.1:39979)
I20260924 19:00:11.007952 2856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34791
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 19:00:11.031978 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40969
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:11.346733 3051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:11.347050 3051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:11.347143 3051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:11.354506 3051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:11.354604 3051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:11.354714 3051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:00:11.364331 3051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:11.365482 3051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:11.366690 3051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:11.379578 3056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:11.379635 3057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:11.379586 3059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:11.380466 3051 server_base.cc:1034] running on GCE node
I20260924 19:00:11.381124 3051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:11.382151 3051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:11.383338 3051 hybrid_clock.cc:648] HybridClock initialized: now 1790276411383276 us; error 63 us; skew 500 ppm
I20260924 19:00:11.383599 3051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:11 dist-test-slave-9fn1 krb5kdc[2811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:11.386523 3051 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 19:00:11.387935 3051 webserver.cc:460] Webserver started at http://127.0.97.2:45723/ using document root <none> and password file <none>
I20260924 19:00:11.388593 3051 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:11.388721 3051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:11.389043 3051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:11.391892 3051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/data/instance:
uuid: "2c74c83213844945bf8972944bceaf6b"
format_stamp: "Formatted at 2026-09-24 19:00:11 on dist-test-slave-9fn1"
server_key: "a8901f946225af40bdeb462f2011f0e6"
server_key_iv: "b4d4bbaacc113384b350c6e46643d1ca"
server_key_version: "encryptionkey@0"
I20260924 19:00:11.392647 3051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2c74c83213844945bf8972944bceaf6b"
format_stamp: "Formatted at 2026-09-24 19:00:11 on dist-test-slave-9fn1"
server_key: "a8901f946225af40bdeb462f2011f0e6"
server_key_iv: "b4d4bbaacc113384b350c6e46643d1ca"
server_key_version: "encryptionkey@0"
I20260924 19:00:11.397622 3051 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260924 19:00:11.400772 3066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:11.402084 3051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 19:00:11.402246 3051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "2c74c83213844945bf8972944bceaf6b"
format_stamp: "Formatted at 2026-09-24 19:00:11 on dist-test-slave-9fn1"
server_key: "a8901f946225af40bdeb462f2011f0e6"
server_key_iv: "b4d4bbaacc113384b350c6e46643d1ca"
server_key_version: "encryptionkey@0"
I20260924 19:00:11.402450 3051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:11.436915 3051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:11.441187 3051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:11.441484 3051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:11.442499 3051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:11.443817 3051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:11.443897 3051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:11.444067 3051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:11.444130 3051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:11.476315 3051 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45719
I20260924 19:00:11.476392 3178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45719 every 8 connection(s)
I20260924 19:00:11.477576 3051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:00:11.479103 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 3051
I20260924 19:00:11.479313 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/wal/instance
I20260924 19:00:11.479990 388 external_mini_cluster.cc:1428] Setting key 82ba35be480f856a97c16c050a3bdacc
Sep 24 19:00:11 dist-test-slave-9fn1 krb5kdc[2811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 19:00:11.500023 3047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:11.480328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48865 (local address 127.0.97.62:40969)
0924 19:00:11.480537 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:11.480543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:11.481957 (+ 1414us) server_negotiation.cc:407] Connection header received
0924 19:00:11.484732 (+ 2775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:11.484735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:11.484792 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:11.484876 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:11.486899 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:11.487599 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:11.489191 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:11.490003 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:11.490139 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:11.493358 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:11.493367 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:11.493374 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:11.493416 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:11.497043 (+ 3627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:11.497803 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:11.497808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:11.497814 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:11.497981 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:11.498394 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:11.498403 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:11.498408 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:11.498777 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:11.498927 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:11.499626 (+ 699us) server_negotiation.cc:299] Negotiation successful
0924 19:00:11.499784 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 19:00:11.501057 3181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:11.480930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40969 (local address 127.0.97.2:48865)
0924 19:00:11.481776 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:11.481834 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:11.484441 (+ 2607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:11.485139 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:11.485152 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:11.485956 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:11.486743 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:11.486767 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:11.487744 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:11.487752 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:11.488986 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:11.488997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:11.490205 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:11.490209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:11.490437 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:11.491091 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:11.491129 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:11.493145 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:11.497183 (+ 4038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:11.497189 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:11.497201 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:11.497693 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:11.498093 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:11.498097 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:11.498099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:11.498257 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:11.498911 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:11.498917 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:11.499284 (+ 367us) client_negotiation.cc:769] Sending connection context
0924 19:00:11.499519 (+ 235us) client_negotiation.cc:240] Negotiation successful
0924 19:00:11.499894 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":350,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260924 19:00:11.502605 3179 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40969
I20260924 19:00:11.502943 3179 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:11.503634 3179 heartbeater.cc:511] Master 127.0.97.62:40969 requested a full tablet report, sending...
I20260924 19:00:11.505205 2856 ts_manager.cc:194] Registered new tserver with Master: 2c74c83213844945bf8972944bceaf6b (127.0.97.2:45719)
I20260924 19:00:11.506093 2856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48865
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 19:00:11.524595 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40969
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:11.834007 3184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:11.834291 3184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:11.834383 3184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:11.841358 3184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:11.841461 3184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:11.841585 3184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:00:11.850983 3184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:11.852341 3184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:11.853629 3184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:11.866626 3189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:11.866590 3190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:11.866846 3192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:11.867611 3184 server_base.cc:1034] running on GCE node
I20260924 19:00:11.868171 3184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:11.868978 3184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:11.870210 3184 hybrid_clock.cc:648] HybridClock initialized: now 1790276411870172 us; error 52 us; skew 500 ppm
I20260924 19:00:11.870512 3184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:11 dist-test-slave-9fn1 krb5kdc[2811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:11.873432 3184 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 19:00:11.874816 3184 webserver.cc:460] Webserver started at http://127.0.97.3:32791/ using document root <none> and password file <none>
I20260924 19:00:11.875463 3184 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:11.875603 3184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:11.875906 3184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:11.878707 3184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/data/instance:
uuid: "db63d62e20674f4f93408ccc308cf2dc"
format_stamp: "Formatted at 2026-09-24 19:00:11 on dist-test-slave-9fn1"
server_key: "b237d31f96c8466e2c80659351118441"
server_key_iv: "586ff7226fb31b437c838aab6fff1fa2"
server_key_version: "encryptionkey@0"
I20260924 19:00:11.879474 3184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/wal/instance:
uuid: "db63d62e20674f4f93408ccc308cf2dc"
format_stamp: "Formatted at 2026-09-24 19:00:11 on dist-test-slave-9fn1"
server_key: "b237d31f96c8466e2c80659351118441"
server_key_iv: "586ff7226fb31b437c838aab6fff1fa2"
server_key_version: "encryptionkey@0"
I20260924 19:00:11.884531 3184 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 19:00:11.887876 3199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:11.889119 3184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:00:11.889304 3184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "db63d62e20674f4f93408ccc308cf2dc"
format_stamp: "Formatted at 2026-09-24 19:00:11 on dist-test-slave-9fn1"
server_key: "b237d31f96c8466e2c80659351118441"
server_key_iv: "586ff7226fb31b437c838aab6fff1fa2"
server_key_version: "encryptionkey@0"
I20260924 19:00:11.889565 3184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:11.904289 3184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:11.909282 3184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:11.909619 3184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:11.910562 3184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:11.911798 3184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:11.911899 3184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:11.912017 3184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:11.912096 3184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:11.945310 3184 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45537
I20260924 19:00:11.945343 3311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45537 every 8 connection(s)
I20260924 19:00:11.946626 3184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:00:11.951334 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 3184
I20260924 19:00:11.951519 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/wal/instance
I20260924 19:00:11.951984 388 external_mini_cluster.cc:1428] Setting key 981df935bce26c4406aa4fb97b3bae6b
Sep 24 19:00:11 dist-test-slave-9fn1 krb5kdc[2811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276411, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:11.965898 3047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:11.949105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37699 (local address 127.0.97.62:40969)
0924 19:00:11.949341 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:11.949348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:11.950494 (+ 1146us) server_negotiation.cc:407] Connection header received
0924 19:00:11.952569 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:11.952573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:11.952635 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:11.952720 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:11.954480 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:11.955424 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:11.956928 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:11.957315 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:11.957442 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:11.960556 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:11.960564 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:11.960573 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:11.960603 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:11.963194 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:11.963828 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:11.963832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:11.963834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:11.963933 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:11.964390 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:11.964393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:11.964395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:11.964617 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:11.964716 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:11.965336 (+ 620us) server_negotiation.cc:299] Negotiation successful
0924 19:00:11.965537 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 19:00:11.967173 3314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:11.949646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40969 (local address 127.0.97.3:37699)
0924 19:00:11.950312 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:11.950353 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:11.952227 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:11.952986 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:11.952994 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:11.953653 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:11.954316 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:11.954333 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:11.955569 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:11.955575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:11.956763 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:11.956774 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:11.957566 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:11.957574 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:11.957800 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:11.958376 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:11.958401 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:11.960369 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:11.963314 (+ 2945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:11.963320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:11.963332 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:11.963720 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:11.964051 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:11.964055 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:11.964057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:11.964293 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:11.964715 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:11.964738 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:11.965013 (+ 275us) client_negotiation.cc:769] Sending connection context
0924 19:00:11.965259 (+ 246us) client_negotiation.cc:240] Negotiation successful
0924 19:00:11.965805 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":258,"threads_started":1}
I20260924 19:00:11.968387 3312 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40969
I20260924 19:00:11.968659 3312 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:11.969304 3312 heartbeater.cc:511] Master 127.0.97.62:40969 requested a full tablet report, sending...
I20260924 19:00:11.970796 2856 ts_manager.cc:194] Registered new tserver with Master: db63d62e20674f4f93408ccc308cf2dc (127.0.97.3:45537)
I20260924 19:00:11.971552 2856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37699
I20260924 19:00:11.983132 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:00:12.011029 3045 heartbeater.cc:503] Master 127.0.97.62:40969 was elected leader, sending a full tablet report...
W20260924 19:00:12.013296 3047 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:00:12.010556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43142 (local address 127.0.97.62:40969)
0924 19:00:12.010731 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:12.010735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:12.011158 (+ 423us) server_negotiation.cc:407] Connection header received
0924 19:00:12.011367 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:12.011371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:12.011434 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:12.011559 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:12.013048 (+ 1489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43142: BlockingRecv error: recv got EOF from 127.0.0.1:43142 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 19:00:12.016170 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 2917
W20260924 19:00:12.019006 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 2917: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:39979: server requires authentication, but client does not have Kerberos credentials available
I20260924 19:00:12.019172 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 2917
I20260924 19:00:12.034515 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 3051
W20260924 19:00:12.036895 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 3051: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:45719: server requires authentication, but client does not have Kerberos credentials available
I20260924 19:00:12.037050 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 3051
I20260924 19:00:12.052423 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 3184
W20260924 19:00:12.055406 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 3184: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:45537: server requires authentication, but client does not have Kerberos credentials available
I20260924 19:00:12.055527 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 3184
I20260924 19:00:12.070505 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 2824
I20260924 19:00:12.071553 2886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:12.171391 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 2824
2026-09-24T19:00:12Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3458 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:00:12 dist-test-slave-9fn1 krb5kdc[3329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:00:12 dist-test-slave-9fn1 krb5kdc[3329](info): set up 2 sockets
Sep 24 19:00:12 dist-test-slave-9fn1 krb5kdc[3329](info): commencing operation
krb5kdc: starting...
W20260924 19:00:13.249801 388 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:00:13 dist-test-slave-9fn1 krb5kdc[3329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:00:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:00:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 19:00:13.434710 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35489
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:13.741258 3342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:13.741580 3342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:13.741750 3342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:13.749352 3342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:00:13.749442 3342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:13.749486 3342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:13.749596 3342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:00:13.749665 3342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:00:13.759063 3342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:13.760238 3342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:13.761520 3342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:13.772753 3348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:13.772828 3347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:13.772866 3350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:13.773599 3342 server_base.cc:1034] running on GCE node
I20260924 19:00:13.774369 3342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:13.775748 3342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:13.776942 3342 hybrid_clock.cc:648] HybridClock initialized: now 1790276413776899 us; error 38 us; skew 500 ppm
I20260924 19:00:13.777278 3342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:13 dist-test-slave-9fn1 krb5kdc[3329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:13.785132 3342 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:00:13.786656 3342 webserver.cc:460] Webserver started at http://127.0.97.62:41005/ using document root <none> and password file <none>
I20260924 19:00:13.787325 3342 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:13.787468 3342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:13.787761 3342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:13.790637 3342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "6f185a27f63d4f6aaf74ff71d2b75b5a"
format_stamp: "Formatted at 2026-09-24 19:00:13 on dist-test-slave-9fn1"
server_key: "0c689aaa77d3798cff89386bb6f74960"
server_key_iv: "24cfb9c6b97ff317a60773db4f80b145"
server_key_version: "encryptionkey@0"
I20260924 19:00:13.791414 3342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "6f185a27f63d4f6aaf74ff71d2b75b5a"
format_stamp: "Formatted at 2026-09-24 19:00:13 on dist-test-slave-9fn1"
server_key: "0c689aaa77d3798cff89386bb6f74960"
server_key_iv: "24cfb9c6b97ff317a60773db4f80b145"
server_key_version: "encryptionkey@0"
I20260924 19:00:13.796684 3342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:00:13.799921 3357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:13.801209 3342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:00:13.801404 3342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "6f185a27f63d4f6aaf74ff71d2b75b5a"
format_stamp: "Formatted at 2026-09-24 19:00:13 on dist-test-slave-9fn1"
server_key: "0c689aaa77d3798cff89386bb6f74960"
server_key_iv: "24cfb9c6b97ff317a60773db4f80b145"
server_key_version: "encryptionkey@0"
I20260924 19:00:13.801657 3342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:13.819461 3342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:13.823547 3342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:13.823833 3342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:13.841897 3342 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35489
I20260924 19:00:13.841915 3408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35489 every 8 connection(s)
I20260924 19:00:13.843250 3342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:00:13.846863 3409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:13.847450 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 3342
I20260924 19:00:13.847641 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:00:13.848441 388 external_mini_cluster.cc:1428] Setting key 2642b0805df953a6d5a312419cdd634a
I20260924 19:00:13.859615 3409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f185a27f63d4f6aaf74ff71d2b75b5a: Bootstrap starting.
Sep 24 19:00:13 dist-test-slave-9fn1 krb5kdc[3329](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:13.863054 3409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f185a27f63d4f6aaf74ff71d2b75b5a: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:13.864271 3409 log.cc:824] T 00000000000000000000000000000000 P 6f185a27f63d4f6aaf74ff71d2b75b5a: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:13.867333 3409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f185a27f63d4f6aaf74ff71d2b75b5a: No bootstrap required, opened a new log
I20260924 19:00:13.870630 3412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:13.850342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59198 (local address 127.0.97.62:35489)
0924 19:00:13.851073 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:13.851091 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:13.851129 (+ 38us) server_negotiation.cc:407] Connection header received
0924 19:00:13.853062 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:13.853084 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:13.853461 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:13.853947 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:13.855415 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:13.856724 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:13.858126 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:13.858786 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:13.858971 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:13.863407 (+ 4436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:13.863418 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:13.863432 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:13.863476 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:13.866652 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:13.867478 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:13.867485 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:13.867495 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:13.867658 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:13.868100 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:13.868104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:13.868106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:13.868588 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:13.868800 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:13.869244 (+ 444us) server_negotiation.cc:299] Negotiation successful
0924 19:00:13.869648 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":311,"threads_started":1}
I20260924 19:00:13.874323 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:00:13.876143 3409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f185a27f63d4f6aaf74ff71d2b75b5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f185a27f63d4f6aaf74ff71d2b75b5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35489 } }
I20260924 19:00:13.876569 3409 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f185a27f63d4f6aaf74ff71d2b75b5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:13.876691 3409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f185a27f63d4f6aaf74ff71d2b75b5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f185a27f63d4f6aaf74ff71d2b75b5a, State: Initialized, Role: FOLLOWER
I20260924 19:00:13.877346 3409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f185a27f63d4f6aaf74ff71d2b75b5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f185a27f63d4f6aaf74ff71d2b75b5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35489 } }
I20260924 19:00:13.877655 3409 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f185a27f63d4f6aaf74ff71d2b75b5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:00:13.877799 3409 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f185a27f63d4f6aaf74ff71d2b75b5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:00:13.877961 3409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f185a27f63d4f6aaf74ff71d2b75b5a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:13.881431 3409 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f185a27f63d4f6aaf74ff71d2b75b5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f185a27f63d4f6aaf74ff71d2b75b5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35489 } }
I20260924 19:00:13.882089 3409 leader_election.cc:302] T 00000000000000000000000000000000 P 6f185a27f63d4f6aaf74ff71d2b75b5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f185a27f63d4f6aaf74ff71d2b75b5a; no voters:
I20260924 19:00:13.882694 3409 leader_election.cc:288] T 00000000000000000000000000000000 P 6f185a27f63d4f6aaf74ff71d2b75b5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:00:13.882848 3415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f185a27f63d4f6aaf74ff71d2b75b5a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:00:13.884089 3415 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f185a27f63d4f6aaf74ff71d2b75b5a [term 1 LEADER]: Becoming Leader. State: Replica: 6f185a27f63d4f6aaf74ff71d2b75b5a, State: Running, Role: LEADER
I20260924 19:00:13.884727 3415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f185a27f63d4f6aaf74ff71d2b75b5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f185a27f63d4f6aaf74ff71d2b75b5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35489 } }
I20260924 19:00:13.884912 3409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f185a27f63d4f6aaf74ff71d2b75b5a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:00:13.891145 3417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f185a27f63d4f6aaf74ff71d2b75b5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f185a27f63d4f6aaf74ff71d2b75b5a. Latest consensus state: current_term: 1 leader_uuid: "6f185a27f63d4f6aaf74ff71d2b75b5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f185a27f63d4f6aaf74ff71d2b75b5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35489 } } }
I20260924 19:00:13.891450 3416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f185a27f63d4f6aaf74ff71d2b75b5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f185a27f63d4f6aaf74ff71d2b75b5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f185a27f63d4f6aaf74ff71d2b75b5a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35489 } } }
I20260924 19:00:13.892038 3416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f185a27f63d4f6aaf74ff71d2b75b5a [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:13.892035 3417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f185a27f63d4f6aaf74ff71d2b75b5a [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:13.902616 3424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:00:13.908720 3424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:00:13.920320 3424 catalog_manager.cc:1347] Generated new cluster ID: 764691c280324e8992dc454ce17469e6
I20260924 19:00:13.920444 3424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:00:13.927136 3412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:13.919324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59204 (local address 127.0.97.62:35489)
0924 19:00:13.919491 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:13.919502 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:13.920262 (+ 760us) server_negotiation.cc:407] Connection header received
0924 19:00:13.920540 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:13.920549 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:13.920633 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:13.920707 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:13.921809 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:13.922552 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:13.923855 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:13.924283 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:13.924418 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:13.925734 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:13.925739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:13.925742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:00:13.926338 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:13.926396 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:00:13.926423 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:13.926542 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:13.926662 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 19:00:13.926754 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":58}
I20260924 19:00:13.945806 3424 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:00:13.947606 3424 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:00:13.966948 3424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f185a27f63d4f6aaf74ff71d2b75b5a: Generated new TSK 0
I20260924 19:00:13.967819 3424 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:00:13.979876 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 3342
I20260924 19:00:13.980939 3404 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:14.070767 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 3342
2026-09-24T19:00:14Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1893 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:00:14 dist-test-slave-9fn1 krb5kdc[3443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:00:14 dist-test-slave-9fn1 krb5kdc[3443](info): set up 2 sockets
Sep 24 19:00:14 dist-test-slave-9fn1 krb5kdc[3443](info): commencing operation
krb5kdc: starting...
W20260924 19:00:15.153879 388 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:00:15 dist-test-slave-9fn1 krb5kdc[3443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:00:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:00:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 19:00:15.341652 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39453
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39453
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:15.653204 3456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:15.653465 3456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:15.653621 3456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:15.660840 3456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:00:15.660934 3456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:15.661043 3456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:15.661151 3456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:00:15.661218 3456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:00:15.671095 3456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:15.672412 3456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:15.673753 3456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:15.685036 3464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:15.685075 3462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:15.685112 3461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:15.685937 3456 server_base.cc:1034] running on GCE node
I20260924 19:00:15.686698 3456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:15.688122 3456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:15.689311 3456 hybrid_clock.cc:648] HybridClock initialized: now 1790276415689258 us; error 62 us; skew 500 ppm
I20260924 19:00:15.689661 3456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:15 dist-test-slave-9fn1 krb5kdc[3443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:15.692639 3456 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:00:15.694101 3456 webserver.cc:460] Webserver started at http://127.0.97.62:35295/ using document root <none> and password file <none>
I20260924 19:00:15.694720 3456 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:15.694816 3456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:15.695060 3456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:15.698014 3456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "8d4a9b73466e469e8d7556831192dcc9"
format_stamp: "Formatted at 2026-09-24 19:00:15 on dist-test-slave-9fn1"
server_key: "158ae507299bdb6e65e72f850d87de6f"
server_key_iv: "e366bc28b55a2a0acde089b496efe81c"
server_key_version: "encryptionkey@0"
I20260924 19:00:15.698701 3456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "8d4a9b73466e469e8d7556831192dcc9"
format_stamp: "Formatted at 2026-09-24 19:00:15 on dist-test-slave-9fn1"
server_key: "158ae507299bdb6e65e72f850d87de6f"
server_key_iv: "e366bc28b55a2a0acde089b496efe81c"
server_key_version: "encryptionkey@0"
I20260924 19:00:15.703652 3456 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:00:15.706957 3471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:15.708271 3456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:00:15.708407 3456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "8d4a9b73466e469e8d7556831192dcc9"
format_stamp: "Formatted at 2026-09-24 19:00:15 on dist-test-slave-9fn1"
server_key: "158ae507299bdb6e65e72f850d87de6f"
server_key_iv: "e366bc28b55a2a0acde089b496efe81c"
server_key_version: "encryptionkey@0"
I20260924 19:00:15.708575 3456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:15.724174 3456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:15.728549 3456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:15.728777 3456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:15.746508 3456 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39453
I20260924 19:00:15.746536 3522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39453 every 8 connection(s)
I20260924 19:00:15.747854 3456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:00:15.748167 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 3456
I20260924 19:00:15.748373 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:00:15.749109 388 external_mini_cluster.cc:1428] Setting key 3fa0cf2d03b1f1444fcd05af27adf445
I20260924 19:00:15.752398 3523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 19:00:15 dist-test-slave-9fn1 krb5kdc[3443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:15.765105 3523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d4a9b73466e469e8d7556831192dcc9: Bootstrap starting.
I20260924 19:00:15.769196 3523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d4a9b73466e469e8d7556831192dcc9: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:15.770507 3523 log.cc:824] T 00000000000000000000000000000000 P 8d4a9b73466e469e8d7556831192dcc9: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:15.773027 3526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:15.751072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54136 (local address 127.0.97.62:39453)
0924 19:00:15.751824 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:15.751839 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:15.751886 (+ 47us) server_negotiation.cc:407] Connection header received
0924 19:00:15.754097 (+ 2211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:15.754120 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:15.754585 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:15.754995 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:15.756162 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:15.758229 (+ 2067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:15.759400 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:15.760401 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:15.760589 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:15.764672 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:15.764683 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:15.764701 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:15.764748 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:15.768597 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:15.769259 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:15.769268 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:15.769276 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:15.769480 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:15.769870 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:15.769875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:15.769878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:15.770549 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:15.770830 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:15.771316 (+ 486us) server_negotiation.cc:299] Negotiation successful
0924 19:00:15.771759 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":317,"threads_started":1}
I20260924 19:00:15.773883 3523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d4a9b73466e469e8d7556831192dcc9: No bootstrap required, opened a new log
I20260924 19:00:15.782249 3523 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d4a9b73466e469e8d7556831192dcc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4a9b73466e469e8d7556831192dcc9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39453 } }
I20260924 19:00:15.782625 3523 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d4a9b73466e469e8d7556831192dcc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:15.782748 3523 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d4a9b73466e469e8d7556831192dcc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d4a9b73466e469e8d7556831192dcc9, State: Initialized, Role: FOLLOWER
I20260924 19:00:15.783383 3523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d4a9b73466e469e8d7556831192dcc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4a9b73466e469e8d7556831192dcc9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39453 } }
I20260924 19:00:15.783607 3523 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d4a9b73466e469e8d7556831192dcc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:00:15.783737 3523 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d4a9b73466e469e8d7556831192dcc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:00:15.783883 3523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d4a9b73466e469e8d7556831192dcc9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:15.786412 3523 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d4a9b73466e469e8d7556831192dcc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4a9b73466e469e8d7556831192dcc9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39453 } }
I20260924 19:00:15.786847 3523 leader_election.cc:302] T 00000000000000000000000000000000 P 8d4a9b73466e469e8d7556831192dcc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d4a9b73466e469e8d7556831192dcc9; no voters:
I20260924 19:00:15.787356 3523 leader_election.cc:288] T 00000000000000000000000000000000 P 8d4a9b73466e469e8d7556831192dcc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:00:15.787490 3528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d4a9b73466e469e8d7556831192dcc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:00:15.788185 3528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d4a9b73466e469e8d7556831192dcc9 [term 1 LEADER]: Becoming Leader. State: Replica: 8d4a9b73466e469e8d7556831192dcc9, State: Running, Role: LEADER
I20260924 19:00:15.788691 3528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d4a9b73466e469e8d7556831192dcc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4a9b73466e469e8d7556831192dcc9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39453 } }
I20260924 19:00:15.789356 3523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d4a9b73466e469e8d7556831192dcc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:00:15.792104 3529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d4a9b73466e469e8d7556831192dcc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d4a9b73466e469e8d7556831192dcc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4a9b73466e469e8d7556831192dcc9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39453 } } }
I20260924 19:00:15.792079 3530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d4a9b73466e469e8d7556831192dcc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d4a9b73466e469e8d7556831192dcc9. Latest consensus state: current_term: 1 leader_uuid: "8d4a9b73466e469e8d7556831192dcc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4a9b73466e469e8d7556831192dcc9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39453 } } }
I20260924 19:00:15.792587 3529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d4a9b73466e469e8d7556831192dcc9 [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:15.792645 3530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d4a9b73466e469e8d7556831192dcc9 [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:15.793823 3536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:00:15.799415 3536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:00:15.807456 3536 catalog_manager.cc:1347] Generated new cluster ID: 2ba69ac2ef974b21975726d44a7a429e
I20260924 19:00:15.807569 3536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:00:15.820209 3536 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:00:15.821981 3536 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:00:15.828897 3536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d4a9b73466e469e8d7556831192dcc9: Generated new TSK 0
I20260924 19:00:15.829784 3536 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 19:00:15.890275 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39453
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:16.212232 3549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:16.212504 3549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:16.212721 3549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:16.219816 3549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:16.219951 3549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:16.220139 3549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:00:16.229481 3549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:16.230727 3549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:16.232049 3549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:16.244823 3557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:16.244853 3554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:16.244901 3555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:16.245786 3549 server_base.cc:1034] running on GCE node
I20260924 19:00:16.246387 3549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:16.247327 3549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:16.248565 3549 hybrid_clock.cc:648] HybridClock initialized: now 1790276416248509 us; error 48 us; skew 500 ppm
I20260924 19:00:16.248891 3549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:16 dist-test-slave-9fn1 krb5kdc[3443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:16.251894 3549 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 19:00:16.253305 3549 webserver.cc:460] Webserver started at http://127.0.97.1:45333/ using document root <none> and password file <none>
I20260924 19:00:16.254062 3549 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:16.254205 3549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:16.254508 3549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:16.257285 3549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/data/instance:
uuid: "cc3e6463069a4629b57bdd95b691b9e6"
format_stamp: "Formatted at 2026-09-24 19:00:16 on dist-test-slave-9fn1"
server_key: "9ecb2a0db3424ebcb1cfeee49fca6288"
server_key_iv: "b1d628fe140a5210068cf2fc55f231b5"
server_key_version: "encryptionkey@0"
I20260924 19:00:16.258118 3549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/wal/instance:
uuid: "cc3e6463069a4629b57bdd95b691b9e6"
format_stamp: "Formatted at 2026-09-24 19:00:16 on dist-test-slave-9fn1"
server_key: "9ecb2a0db3424ebcb1cfeee49fca6288"
server_key_iv: "b1d628fe140a5210068cf2fc55f231b5"
server_key_version: "encryptionkey@0"
I20260924 19:00:16.263381 3549 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 19:00:16.266834 3564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:16.268128 3549 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:00:16.268301 3549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "cc3e6463069a4629b57bdd95b691b9e6"
format_stamp: "Formatted at 2026-09-24 19:00:16 on dist-test-slave-9fn1"
server_key: "9ecb2a0db3424ebcb1cfeee49fca6288"
server_key_iv: "b1d628fe140a5210068cf2fc55f231b5"
server_key_version: "encryptionkey@0"
I20260924 19:00:16.268520 3549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:16.291983 3549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:16.296052 3549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:16.296339 3549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:16.297236 3549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:16.298539 3549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:16.298637 3549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:16.298779 3549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:16.298846 3549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:16.331303 3549 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37945
I20260924 19:00:16.331332 3676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37945 every 8 connection(s)
I20260924 19:00:16.332593 3549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:00:16.340458 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 3549
I20260924 19:00:16.340667 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-0/wal/instance
I20260924 19:00:16.341348 388 external_mini_cluster.cc:1428] Setting key b4e10027996864969be5c4ceb5e048a2
Sep 24 19:00:16 dist-test-slave-9fn1 krb5kdc[3443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:16.352586 3679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:16.335087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39405 (local address 127.0.97.62:39453)
0924 19:00:16.335502 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:16.335506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:16.336472 (+ 966us) server_negotiation.cc:407] Connection header received
0924 19:00:16.338259 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:16.338263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:16.338325 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:16.338420 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:16.339790 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:16.340830 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:16.342567 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:16.343018 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:16.343154 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:16.347157 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:16.347162 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:16.347166 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:16.347194 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:16.349722 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:16.350422 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:16.350434 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:16.350437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:16.350540 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:16.350948 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:16.350952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:16.350955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:16.351209 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:16.351367 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:16.352014 (+ 647us) server_negotiation.cc:299] Negotiation successful
0924 19:00:16.352227 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":210,"threads_started":1}
I20260924 19:00:16.353586 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:16.335647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39453 (local address 127.0.97.1:39405)
0924 19:00:16.336314 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:16.336359 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:16.338021 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:16.338615 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:16.338623 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:16.339113 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:16.339646 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:16.339662 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:16.340989 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:16.340993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:16.342418 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:16.342429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:16.343261 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:16.343269 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:16.343559 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:16.344273 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:16.344306 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:16.346953 (+ 2647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:16.349873 (+ 2920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:16.349882 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:16.349898 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:16.350296 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:16.350647 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:16.350653 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:16.350657 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:16.350827 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:16.351342 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:16.351350 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:16.351660 (+ 310us) client_negotiation.cc:769] Sending connection context
0924 19:00:16.351910 (+ 250us) client_negotiation.cc:240] Negotiation successful
0924 19:00:16.352417 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":259,"threads_started":1}
I20260924 19:00:16.355262 3677 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39453
I20260924 19:00:16.355576 3677 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:16.356253 3677 heartbeater.cc:511] Master 127.0.97.62:39453 requested a full tablet report, sending...
I20260924 19:00:16.358681 3488 ts_manager.cc:194] Registered new tserver with Master: cc3e6463069a4629b57bdd95b691b9e6 (127.0.97.1:37945)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 19:00:16.360352 3488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 19:00:16.384449 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39453
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:16.701882 3683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:16.702147 3683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:16.702315 3683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:16.709085 3683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:16.709215 3683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:16.709396 3683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:00:16.718940 3683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:16.720202 3683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:16.721534 3683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:16.733762 3689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:16.733903 3688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:16.733975 3691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:16.735287 3683 server_base.cc:1034] running on GCE node
I20260924 19:00:16.735940 3683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:16.736799 3683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:16.738029 3683 hybrid_clock.cc:648] HybridClock initialized: now 1790276416737984 us; error 62 us; skew 500 ppm
I20260924 19:00:16.738351 3683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:16 dist-test-slave-9fn1 krb5kdc[3443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:16.741302 3683 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 19:00:16.742692 3683 webserver.cc:460] Webserver started at http://127.0.97.2:40365/ using document root <none> and password file <none>
I20260924 19:00:16.743394 3683 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:16.743530 3683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:16.743820 3683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:16.746683 3683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/data/instance:
uuid: "1dca65f2f5964c0c843f1d5daf1e301d"
format_stamp: "Formatted at 2026-09-24 19:00:16 on dist-test-slave-9fn1"
server_key: "6a0060e0e280f0a39eb55063935034cb"
server_key_iv: "7ce76da50412caa48d06ce36e285a2a4"
server_key_version: "encryptionkey@0"
I20260924 19:00:16.747450 3683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1dca65f2f5964c0c843f1d5daf1e301d"
format_stamp: "Formatted at 2026-09-24 19:00:16 on dist-test-slave-9fn1"
server_key: "6a0060e0e280f0a39eb55063935034cb"
server_key_iv: "7ce76da50412caa48d06ce36e285a2a4"
server_key_version: "encryptionkey@0"
I20260924 19:00:16.752419 3683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:00:16.755884 3698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:16.757212 3683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 19:00:16.757385 3683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "1dca65f2f5964c0c843f1d5daf1e301d"
format_stamp: "Formatted at 2026-09-24 19:00:16 on dist-test-slave-9fn1"
server_key: "6a0060e0e280f0a39eb55063935034cb"
server_key_iv: "7ce76da50412caa48d06ce36e285a2a4"
server_key_version: "encryptionkey@0"
I20260924 19:00:16.757611 3683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:16.787204 3683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:16.791278 3683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:16.791575 3683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:16.792588 3683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:16.793968 3683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:16.794078 3683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:16.794215 3683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:16.794308 3683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:16.826742 3683 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36013
I20260924 19:00:16.826794 3810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36013 every 8 connection(s)
I20260924 19:00:16.828053 3683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:00:16.831506 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 3683
I20260924 19:00:16.831691 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-1/wal/instance
I20260924 19:00:16.832183 388 external_mini_cluster.cc:1428] Setting key 402a4acac8aada89b49f7a49b97a1ee1
Sep 24 19:00:16 dist-test-slave-9fn1 krb5kdc[3443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:16.849042 3679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:16.830640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58443 (local address 127.0.97.62:39453)
0924 19:00:16.830846 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:16.830852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:16.832217 (+ 1365us) server_negotiation.cc:407] Connection header received
0924 19:00:16.834307 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:16.834314 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:16.834393 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:16.834503 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:16.835960 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:16.837087 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:16.838649 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:16.839310 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:16.839408 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:16.843520 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:16.843525 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:16.843528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:16.843558 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:16.846049 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:16.846840 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:16.846852 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:16.846856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:16.846979 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:16.847410 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:16.847415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:16.847422 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:16.847670 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:16.847821 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:16.848481 (+ 660us) server_negotiation.cc:299] Negotiation successful
0924 19:00:16.848693 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 19:00:16.850224 3813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:16.831213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39453 (local address 127.0.97.2:58443)
0924 19:00:16.832016 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:16.832074 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:16.833998 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:16.834710 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:16.834717 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:16.835224 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:16.835790 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:16.835807 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:16.837225 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:16.837228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:16.838510 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:16.838521 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:16.839529 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:16.839536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:16.839903 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:16.840631 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:16.840665 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:16.843287 (+ 2622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:16.846192 (+ 2905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:16.846203 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:16.846220 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:16.846705 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:16.847091 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:16.847097 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:16.847101 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:16.847289 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:16.847796 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:16.847804 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:16.848137 (+ 333us) client_negotiation.cc:769] Sending connection context
0924 19:00:16.848387 (+ 250us) client_negotiation.cc:240] Negotiation successful
0924 19:00:16.848923 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":346,"threads_started":1}
I20260924 19:00:16.851616 3811 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39453
I20260924 19:00:16.851917 3811 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260924 19:00:16.852703 3811 heartbeater.cc:511] Master 127.0.97.62:39453 requested a full tablet report, sending...
I20260924 19:00:16.854321 3488 ts_manager.cc:194] Registered new tserver with Master: 1dca65f2f5964c0c843f1d5daf1e301d (127.0.97.2:36013)
I20260924 19:00:16.855123 3488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 19:00:16.876587 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39453
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:17.195335 3816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:17.195611 3816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:17.195712 3816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:17.203202 3816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:17.203315 3816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:17.203439 3816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:00:17.213037 3816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39453
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:17.214254 3816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:17.215564 3816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:17.228773 3822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:17.228789 3824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:17.228591 3821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:17.229718 3816 server_base.cc:1034] running on GCE node
I20260924 19:00:17.230429 3816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:17.231448 3816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:17.232707 3816 hybrid_clock.cc:648] HybridClock initialized: now 1790276417232661 us; error 50 us; skew 500 ppm
I20260924 19:00:17.233057 3816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:17 dist-test-slave-9fn1 krb5kdc[3443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:17.236495 3816 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 19:00:17.238112 3816 webserver.cc:460] Webserver started at http://127.0.97.3:46471/ using document root <none> and password file <none>
I20260924 19:00:17.238847 3816 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:17.239003 3816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:17.239326 3816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:17.242488 3816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/data/instance:
uuid: "894ec9c0b70a45f1b1418876c7ed9531"
format_stamp: "Formatted at 2026-09-24 19:00:17 on dist-test-slave-9fn1"
server_key: "94fc20ebccc2b792fe6a10e5ea22342f"
server_key_iv: "9576153b061f5d7a8c23356b7edb9197"
server_key_version: "encryptionkey@0"
I20260924 19:00:17.243394 3816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/wal/instance:
uuid: "894ec9c0b70a45f1b1418876c7ed9531"
format_stamp: "Formatted at 2026-09-24 19:00:17 on dist-test-slave-9fn1"
server_key: "94fc20ebccc2b792fe6a10e5ea22342f"
server_key_iv: "9576153b061f5d7a8c23356b7edb9197"
server_key_version: "encryptionkey@0"
I20260924 19:00:17.248793 3816 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260924 19:00:17.252473 3831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:17.253909 3816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:00:17.254078 3816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "894ec9c0b70a45f1b1418876c7ed9531"
format_stamp: "Formatted at 2026-09-24 19:00:17 on dist-test-slave-9fn1"
server_key: "94fc20ebccc2b792fe6a10e5ea22342f"
server_key_iv: "9576153b061f5d7a8c23356b7edb9197"
server_key_version: "encryptionkey@0"
I20260924 19:00:17.254281 3816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:17.301702 3816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:17.307492 3816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:17.307864 3816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:17.309134 3816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:17.310945 3816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:17.311043 3816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:17.311169 3816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:17.311231 3816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:17.347936 3816 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39901
I20260924 19:00:17.347987 3943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39901 every 8 connection(s)
I20260924 19:00:17.349713 3816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:00:17.354667 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 3816
I20260924 19:00:17.354857 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276386554690-388-0/minicluster-data/ts-2/wal/instance
I20260924 19:00:17.355427 388 external_mini_cluster.cc:1428] Setting key bed60ac1e6e89db8d4403acfc0081e05
I20260924 19:00:17.363750 3677 heartbeater.cc:503] Master 127.0.97.62:39453 was elected leader, sending a full tablet report...
Sep 24 19:00:17 dist-test-slave-9fn1 krb5kdc[3443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:17.370659 3946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:17.352786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34397 (local address 127.0.97.62:39453)
0924 19:00:17.353295 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:17.353302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:17.356895 (+ 3593us) server_negotiation.cc:407] Connection header received
0924 19:00:17.356978 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:17.356983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:17.357065 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:17.357178 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:17.359103 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:17.359884 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:17.361444 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:17.361970 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:17.362091 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:17.365301 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:17.365306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:17.365309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:17.365338 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:17.367775 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:17.368415 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:17.368418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:17.368421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:17.368527 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:17.368966 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:17.368969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:17.368971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:17.369203 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:17.369369 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:17.370086 (+ 717us) server_negotiation.cc:299] Negotiation successful
0924 19:00:17.370303 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":292,"threads_started":1}
I20260924 19:00:17.371711 3947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:17.353507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39453 (local address 127.0.97.3:34397)
0924 19:00:17.354246 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:17.354296 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:17.356534 (+ 2238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:17.357449 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:17.357458 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:17.358238 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:17.358966 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:17.358983 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:17.360004 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:17.360008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:17.361326 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:17.361342 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:17.362171 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:17.362176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:17.362359 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:17.362923 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:17.362943 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:17.365039 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:17.367897 (+ 2858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:17.367904 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:17.367916 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:17.368317 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:17.368614 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:17.368617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:17.368620 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:17.368823 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:17.369333 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:17.369340 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:17.369750 (+ 410us) client_negotiation.cc:769] Sending connection context
0924 19:00:17.370010 (+ 260us) client_negotiation.cc:240] Negotiation successful
0924 19:00:17.370502 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":282,"threads_started":1}
I20260924 19:00:17.372906 3944 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39453
I20260924 19:00:17.373199 3944 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:17.373947 3944 heartbeater.cc:511] Master 127.0.97.62:39453 requested a full tablet report, sending...
I20260924 19:00:17.375167 3488 ts_manager.cc:194] Registered new tserver with Master: 894ec9c0b70a45f1b1418876c7ed9531 (127.0.97.3:39901)
I20260924 19:00:17.375898 3488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34397
I20260924 19:00:17.382759 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 19:00:17 dist-test-slave-9fn1 krb5kdc[3443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276417, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 19:00:17 dist-test-slave-9fn1 krb5kdc[3443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276417, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:17.427505 3946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:17.414772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54152 (local address 127.0.97.62:39453)
0924 19:00:17.415015 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:17.415021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:17.415307 (+ 286us) server_negotiation.cc:407] Connection header received
0924 19:00:17.415459 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:17.415463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:17.415539 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:17.415623 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:17.416880 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:17.417651 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:17.418867 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:17.419256 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:17.419348 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:17.422995 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:17.422999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:17.423002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:17.423031 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:17.425254 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:17.425904 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:17.425907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:17.425910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:17.426022 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:17.426327 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:17.426330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:17.426332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:17.426620 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:17.426797 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:17.427037 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 19:00:17.427188 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260924 19:00:17.428067 3488 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54152
I20260924 19:00:17.429886 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 3549
Sep 24 19:00:17 dist-test-slave-9fn1 krb5kdc[3443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276417, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 19:00:17.443974 3956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:17.430734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51878 (local address 127.0.97.1:37945)
0924 19:00:17.431296 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:17.431303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:17.431327 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:00:17.431378 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:17.431383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:17.431580 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:17.431702 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:17.432706 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:17.433744 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:17.434852 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:17.435383 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:17.435474 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:17.439187 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:17.439198 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:17.439210 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:17.439237 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:17.441803 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:17.442381 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:17.442388 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:17.442390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:17.442496 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:17.442783 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:17.442786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:17.442788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:17.443082 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:17.443300 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:17.443510 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 19:00:17.443681 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":264,"threads_started":1}
W20260924 19:00:17.445161 3672 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51878
W20260924 19:00:17.446269 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 3549: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:00:17.446476 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 3549
I20260924 19:00:17.461869 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 3683
Sep 24 19:00:17 dist-test-slave-9fn1 krb5kdc[3443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276417, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 19:00:17.476572 3957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:17.462692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33590 (local address 127.0.97.2:36013)
0924 19:00:17.463261 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:17.463268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:17.463294 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:00:17.463354 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:17.463360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:17.463569 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:17.463718 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:17.464783 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:17.465904 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:17.467087 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:17.467588 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:17.467698 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:17.471546 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:17.471555 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:17.471568 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:17.471595 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:17.474231 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:17.474797 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:17.474803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:17.474806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:17.474913 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:17.475207 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:17.475210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:17.475212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:17.475530 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:17.475747 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:17.475999 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 19:00:17.476182 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":318,"threads_started":1}
W20260924 19:00:17.477852 3806 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33590
W20260924 19:00:17.478911 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 3683: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:00:17.479061 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 3683
I20260924 19:00:17.494468 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 3816
Sep 24 19:00:17 dist-test-slave-9fn1 krb5kdc[3443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276417, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 19:00:17.510031 3958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:17.495607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38472 (local address 127.0.97.3:39901)
0924 19:00:17.496296 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:17.496302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:17.496330 (+ 28us) server_negotiation.cc:407] Connection header received
0924 19:00:17.496391 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:17.496397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:17.496600 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:17.496725 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:17.498021 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:17.499019 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:17.500174 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:17.500706 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:17.500800 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:17.504750 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:17.504762 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:17.504777 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:17.504807 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:17.507621 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:17.508207 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:17.508214 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:17.508216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:17.508327 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:17.508607 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:17.508610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:17.508612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:17.508932 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:17.509154 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:17.509339 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 19:00:17.509649 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":390,"threads_started":1}
W20260924 19:00:17.511237 3939 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38472
W20260924 19:00:17.512390 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 3816: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:00:17.512594 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 3816
I20260924 19:00:17.527927 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 3456
I20260924 19:00:17.529091 3518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:17.629206 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 3456
2026-09-24T19:00:17Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3564 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:00:17 dist-test-slave-9fn1 krb5kdc[3962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:00:17 dist-test-slave-9fn1 krb5kdc[3962](info): set up 2 sockets
Sep 24 19:00:17 dist-test-slave-9fn1 krb5kdc[3962](info): commencing operation
krb5kdc: starting...
W20260924 19:00:18.710101 388 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:00:18 dist-test-slave-9fn1 krb5kdc[3962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:00:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:00:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 19:00:18.895327 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43171
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:19.210508 3975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:19.210793 3975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:19.210981 3975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:19.218031 3975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:00:19.218132 3975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:19.218242 3975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:19.218341 3975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:00:19.218423 3975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:00:19.227962 3975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:19.229218 3975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:19.230504 3975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:19.241916 3980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:19.242017 3981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:19.242609 3983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:19.243134 3975 server_base.cc:1034] running on GCE node
I20260924 19:00:19.243950 3975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:19.245324 3975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:19.246583 3975 hybrid_clock.cc:648] HybridClock initialized: now 1790276419246521 us; error 52 us; skew 500 ppm
I20260924 19:00:19.246948 3975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:19 dist-test-slave-9fn1 krb5kdc[3962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:19.253417 3975 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:00:19.255072 3975 webserver.cc:460] Webserver started at http://127.0.97.62:34933/ using document root <none> and password file <none>
I20260924 19:00:19.255725 3975 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:19.255823 3975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:19.256090 3975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:19.259421 3975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "2b20290da01048ac8325446f905bf83e"
format_stamp: "Formatted at 2026-09-24 19:00:19 on dist-test-slave-9fn1"
server_key: "ec87dbe0a2ec93320bf78219b6250150"
server_key_iv: "28efd27d1eb6f4e4d0bee2b2e47debbc"
server_key_version: "encryptionkey@0"
I20260924 19:00:19.260280 3975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "2b20290da01048ac8325446f905bf83e"
format_stamp: "Formatted at 2026-09-24 19:00:19 on dist-test-slave-9fn1"
server_key: "ec87dbe0a2ec93320bf78219b6250150"
server_key_iv: "28efd27d1eb6f4e4d0bee2b2e47debbc"
server_key_version: "encryptionkey@0"
I20260924 19:00:19.265342 3975 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 19:00:19.268612 3990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:19.270073 3975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:00:19.270244 3975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "2b20290da01048ac8325446f905bf83e"
format_stamp: "Formatted at 2026-09-24 19:00:19 on dist-test-slave-9fn1"
server_key: "ec87dbe0a2ec93320bf78219b6250150"
server_key_iv: "28efd27d1eb6f4e4d0bee2b2e47debbc"
server_key_version: "encryptionkey@0"
I20260924 19:00:19.270439 3975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:19.291152 3975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:19.295073 3975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:19.295363 3975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:19.313391 3975 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43171
I20260924 19:00:19.313436 4041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43171 every 8 connection(s)
I20260924 19:00:19.314818 3975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:00:19.318387 4042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:19.323311 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 3975
I20260924 19:00:19.323535 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:00:19.324249 388 external_mini_cluster.cc:1428] Setting key c6adf1ca88c6b91821dda8339c0f2b7a
I20260924 19:00:19.329761 4042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b20290da01048ac8325446f905bf83e: Bootstrap starting.
I20260924 19:00:19.333485 4042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b20290da01048ac8325446f905bf83e: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:19.334744 4042 log.cc:824] T 00000000000000000000000000000000 P 2b20290da01048ac8325446f905bf83e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:19.337739 4042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b20290da01048ac8325446f905bf83e: No bootstrap required, opened a new log
Sep 24 19:00:19 dist-test-slave-9fn1 krb5kdc[3962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:19.348206 4045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:19.326342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36128 (local address 127.0.97.62:43171)
0924 19:00:19.327162 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:19.327173 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:19.327219 (+ 46us) server_negotiation.cc:407] Connection header received
0924 19:00:19.328908 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:19.328931 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:19.329323 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:19.329730 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:19.331449 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:19.332729 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:19.334520 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:19.335339 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:19.335549 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:19.339438 (+ 3889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:19.339452 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:19.339476 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:19.339521 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:19.343622 (+ 4101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:19.344270 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:19.344283 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:19.344291 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:19.344490 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:19.344820 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:19.344825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:19.344829 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:19.345559 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:19.345837 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:19.346318 (+ 481us) server_negotiation.cc:299] Negotiation successful
0924 19:00:19.346816 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":393,"threads_started":1}
I20260924 19:00:19.348519 4042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b20290da01048ac8325446f905bf83e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b20290da01048ac8325446f905bf83e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43171 } }
I20260924 19:00:19.348904 4042 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b20290da01048ac8325446f905bf83e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:19.349014 4042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b20290da01048ac8325446f905bf83e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b20290da01048ac8325446f905bf83e, State: Initialized, Role: FOLLOWER
I20260924 19:00:19.349737 4042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b20290da01048ac8325446f905bf83e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b20290da01048ac8325446f905bf83e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43171 } }
I20260924 19:00:19.349992 4042 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b20290da01048ac8325446f905bf83e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:00:19.350152 4042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b20290da01048ac8325446f905bf83e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:00:19.350342 4042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b20290da01048ac8325446f905bf83e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:19.352517 4042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b20290da01048ac8325446f905bf83e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b20290da01048ac8325446f905bf83e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43171 } }
I20260924 19:00:19.352994 4042 leader_election.cc:302] T 00000000000000000000000000000000 P 2b20290da01048ac8325446f905bf83e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b20290da01048ac8325446f905bf83e; no voters:
I20260924 19:00:19.353636 4042 leader_election.cc:288] T 00000000000000000000000000000000 P 2b20290da01048ac8325446f905bf83e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:00:19.353761 4047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b20290da01048ac8325446f905bf83e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:00:19.354421 4047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b20290da01048ac8325446f905bf83e [term 1 LEADER]: Becoming Leader. State: Replica: 2b20290da01048ac8325446f905bf83e, State: Running, Role: LEADER
I20260924 19:00:19.354977 4047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b20290da01048ac8325446f905bf83e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b20290da01048ac8325446f905bf83e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43171 } }
I20260924 19:00:19.355746 4042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b20290da01048ac8325446f905bf83e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:00:19.357911 4049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b20290da01048ac8325446f905bf83e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b20290da01048ac8325446f905bf83e. Latest consensus state: current_term: 1 leader_uuid: "2b20290da01048ac8325446f905bf83e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b20290da01048ac8325446f905bf83e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43171 } } }
I20260924 19:00:19.357861 4048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b20290da01048ac8325446f905bf83e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b20290da01048ac8325446f905bf83e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b20290da01048ac8325446f905bf83e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43171 } } }
I20260924 19:00:19.358321 4048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b20290da01048ac8325446f905bf83e [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:19.363734 4049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b20290da01048ac8325446f905bf83e [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:19.365772 4054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:00:19.370992 4054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:00:19.379575 4054 catalog_manager.cc:1347] Generated new cluster ID: 32d197fa0ed2470daa4d834a77f21948
I20260924 19:00:19.379678 4054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:00:19.390141 4054 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:00:19.391587 4054 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:00:19.401796 4054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b20290da01048ac8325446f905bf83e: Generated new TSK 0
I20260924 19:00:19.402534 4054 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 19:00:19.454957 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43171
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:19.773154 4068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:19.773456 4068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:19.773595 4068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:19.781071 4068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:19.781164 4068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:19.781267 4068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:00:19.790519 4068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:19.791923 4068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:19.793141 4068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:19.806387 4073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:19.806373 4074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:19.806361 4076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:19.807271 4068 server_base.cc:1034] running on GCE node
I20260924 19:00:19.807840 4068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:19.808734 4068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:19.809949 4068 hybrid_clock.cc:648] HybridClock initialized: now 1790276419809907 us; error 53 us; skew 500 ppm
I20260924 19:00:19.810281 4068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:19 dist-test-slave-9fn1 krb5kdc[3962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:19.813265 4068 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 19:00:19.814634 4068 webserver.cc:460] Webserver started at http://127.0.97.1:37977/ using document root <none> and password file <none>
I20260924 19:00:19.815280 4068 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:19.815425 4068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:19.815716 4068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:19.818481 4068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-0/data/instance:
uuid: "c1fd6c0284da4e7490cd12bc1a9fa69e"
format_stamp: "Formatted at 2026-09-24 19:00:19 on dist-test-slave-9fn1"
server_key: "f4ede6e6bc7628470e35b0fcc0cb9bbe"
server_key_iv: "3836eb81374437fe9283ae8098d5610e"
server_key_version: "encryptionkey@0"
I20260924 19:00:19.819237 4068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c1fd6c0284da4e7490cd12bc1a9fa69e"
format_stamp: "Formatted at 2026-09-24 19:00:19 on dist-test-slave-9fn1"
server_key: "f4ede6e6bc7628470e35b0fcc0cb9bbe"
server_key_iv: "3836eb81374437fe9283ae8098d5610e"
server_key_version: "encryptionkey@0"
I20260924 19:00:19.824156 4068 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:00:19.827309 4083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:19.828547 4068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 19:00:19.828727 4068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "c1fd6c0284da4e7490cd12bc1a9fa69e"
format_stamp: "Formatted at 2026-09-24 19:00:19 on dist-test-slave-9fn1"
server_key: "f4ede6e6bc7628470e35b0fcc0cb9bbe"
server_key_iv: "3836eb81374437fe9283ae8098d5610e"
server_key_version: "encryptionkey@0"
I20260924 19:00:19.828940 4068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:19.839582 4068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:19.843142 4068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:19.843434 4068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:19.844272 4068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:19.845467 4068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:19.845649 4068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:19.845786 4068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:19.845855 4068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:19.878321 4068 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39055
I20260924 19:00:19.878412 4195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39055 every 8 connection(s)
I20260924 19:00:19.879611 4068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:00:19.882208 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 4068
I20260924 19:00:19.882412 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-0/wal/instance
I20260924 19:00:19.883179 388 external_mini_cluster.cc:1428] Setting key dec7cccc965c026d241f9ad6eae1b194
Sep 24 19:00:19 dist-test-slave-9fn1 krb5kdc[3962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:19.901238 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:19.882469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40377 (local address 127.0.97.62:43171)
0924 19:00:19.882935 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:19.882942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:19.883478 (+ 536us) server_negotiation.cc:407] Connection header received
0924 19:00:19.885463 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:19.885468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:19.885584 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:19.885693 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:19.887148 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:19.888379 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:19.890032 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:19.890442 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:19.890576 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:19.894424 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:19.894433 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:19.894441 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:19.894482 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:19.898102 (+ 3620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:19.898773 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:19.898781 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:19.898788 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:19.898935 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:19.899390 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:19.899398 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:19.899403 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:19.899813 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:19.899983 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:19.900552 (+ 569us) server_negotiation.cc:299] Negotiation successful
0924 19:00:19.900787 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":230,"threads_started":1}
I20260924 19:00:19.902076 4199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:19.882419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43171 (local address 127.0.97.1:40377)
0924 19:00:19.883308 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:19.883360 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:19.885210 (+ 1850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:19.885911 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:19.885918 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:19.886399 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:19.886981 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:19.886997 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:19.888543 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:19.888547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:19.889869 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:19.889885 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:19.890670 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:19.890679 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:19.890910 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:19.891627 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:19.891658 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:19.894223 (+ 2565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:19.898248 (+ 4025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:19.898254 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:19.898266 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:19.898655 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:19.899071 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:19.899075 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:19.899077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:19.899265 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:19.899937 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:19.899942 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:19.900217 (+ 275us) client_negotiation.cc:769] Sending connection context
0924 19:00:19.900454 (+ 237us) client_negotiation.cc:240] Negotiation successful
0924 19:00:19.900865 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":412,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 19:00:19.903589 4196 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43171
I20260924 19:00:19.903889 4196 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:19.904558 4196 heartbeater.cc:511] Master 127.0.97.62:43171 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260924 19:00:19.906781 4007 ts_manager.cc:194] Registered new tserver with Master: c1fd6c0284da4e7490cd12bc1a9fa69e (127.0.97.1:39055)
I20260924 19:00:19.908840 4007 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 19:00:19.928141 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43171
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:20.241796 4202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:20.242054 4202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:20.242159 4202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:20.249125 4202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:20.249222 4202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:20.249329 4202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:00:20.258641 4202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:20.259794 4202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:20.260994 4202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:20.273443 4207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:20.273520 4210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:20.273988 4208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:20.274950 4202 server_base.cc:1034] running on GCE node
I20260924 19:00:20.275599 4202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:20.276475 4202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:20.277678 4202 hybrid_clock.cc:648] HybridClock initialized: now 1790276420277629 us; error 62 us; skew 500 ppm
I20260924 19:00:20.277978 4202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:20 dist-test-slave-9fn1 krb5kdc[3962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:20.280954 4202 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 19:00:20.282335 4202 webserver.cc:460] Webserver started at http://127.0.97.2:38297/ using document root <none> and password file <none>
I20260924 19:00:20.282984 4202 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:20.283125 4202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:20.283455 4202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:20.286754 4202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-1/data/instance:
uuid: "16c83508c20746cdb31c5d74c4e6379c"
format_stamp: "Formatted at 2026-09-24 19:00:20 on dist-test-slave-9fn1"
server_key: "f72961ba50a6edf216ef77250379e0c4"
server_key_iv: "f6ba0d0e02621293e16d04f10ee0ef38"
server_key_version: "encryptionkey@0"
I20260924 19:00:20.287508 4202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-1/wal/instance:
uuid: "16c83508c20746cdb31c5d74c4e6379c"
format_stamp: "Formatted at 2026-09-24 19:00:20 on dist-test-slave-9fn1"
server_key: "f72961ba50a6edf216ef77250379e0c4"
server_key_iv: "f6ba0d0e02621293e16d04f10ee0ef38"
server_key_version: "encryptionkey@0"
I20260924 19:00:20.292259 4202 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 19:00:20.295567 4217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:20.296789 4202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:00:20.296957 4202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "16c83508c20746cdb31c5d74c4e6379c"
format_stamp: "Formatted at 2026-09-24 19:00:20 on dist-test-slave-9fn1"
server_key: "f72961ba50a6edf216ef77250379e0c4"
server_key_iv: "f6ba0d0e02621293e16d04f10ee0ef38"
server_key_version: "encryptionkey@0"
I20260924 19:00:20.297154 4202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:20.332294 4202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:20.336395 4202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:20.336685 4202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:20.337697 4202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:20.338971 4202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:20.339071 4202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:20.339224 4202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:20.339308 4202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:20.372864 4202 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36759
I20260924 19:00:20.372965 4329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36759 every 8 connection(s)
I20260924 19:00:20.374209 4202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:00:20.375979 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 4202
I20260924 19:00:20.376157 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-1/wal/instance
I20260924 19:00:20.376560 388 external_mini_cluster.cc:1428] Setting key dd034b907a8cc7d83cc55d0f2953caee
Sep 24 19:00:20 dist-test-slave-9fn1 krb5kdc[3962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 19:00:20.396903 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:20.376770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58609 (local address 127.0.97.62:43171)
0924 19:00:20.376980 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:20.376986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:20.378461 (+ 1475us) server_negotiation.cc:407] Connection header received
0924 19:00:20.380378 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:20.380386 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:20.380475 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:20.380598 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:20.382043 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:20.383205 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:20.385391 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:20.385874 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:20.385976 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:20.389941 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:20.389950 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:20.389958 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:20.390000 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:20.393755 (+ 3755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:20.394414 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:20.394422 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:20.394427 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:20.394579 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:20.395127 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:20.395133 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:20.395137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:20.395514 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:20.395672 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:20.396313 (+ 641us) server_negotiation.cc:299] Negotiation successful
0924 19:00:20.396558 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 19:00:20.397850 4332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:20.377329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43171 (local address 127.0.97.2:58609)
0924 19:00:20.378195 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:20.378248 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:20.380116 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:20.380784 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:20.380792 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:20.381258 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:20.381888 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:20.381905 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:20.383367 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:20.383371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:20.385225 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:20.385242 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:20.386091 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:20.386101 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:20.386437 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:20.387142 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:20.387179 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:20.389745 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:20.393890 (+ 4145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:20.393897 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:20.393908 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:20.394281 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:20.394705 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:20.394742 (+ 37us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:20.394745 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:20.394992 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:20.395636 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:20.395642 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:20.395945 (+ 303us) client_negotiation.cc:769] Sending connection context
0924 19:00:20.396170 (+ 225us) client_negotiation.cc:240] Negotiation successful
0924 19:00:20.396610 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":379,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260924 19:00:20.399251 4330 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43171
I20260924 19:00:20.399544 4330 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:20.400267 4330 heartbeater.cc:511] Master 127.0.97.62:43171 requested a full tablet report, sending...
I20260924 19:00:20.401844 4007 ts_manager.cc:194] Registered new tserver with Master: 16c83508c20746cdb31c5d74c4e6379c (127.0.97.2:36759)
I20260924 19:00:20.403075 4007 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 19:00:20.422096 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43171
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:20.736984 4335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:20.737296 4335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:20.737457 4335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:20.744693 4335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:20.744840 4335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:20.745004 4335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:00:20.754525 4335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:20.755810 4335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:20.757164 4335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:20.770306 4340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:20.770355 4341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:20.770490 4343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:20.774855 4335 server_base.cc:1034] running on GCE node
I20260924 19:00:20.775430 4335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:20.776274 4335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:20.777487 4335 hybrid_clock.cc:648] HybridClock initialized: now 1790276420777448 us; error 35 us; skew 500 ppm
I20260924 19:00:20.777822 4335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:20 dist-test-slave-9fn1 krb5kdc[3962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:20.780769 4335 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 19:00:20.782210 4335 webserver.cc:460] Webserver started at http://127.0.97.3:33557/ using document root <none> and password file <none>
I20260924 19:00:20.782902 4335 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:20.783047 4335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:20.783346 4335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:20.786266 4335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-2/data/instance:
uuid: "89d8a7e36bbe409aa14cb4eff3c909d2"
format_stamp: "Formatted at 2026-09-24 19:00:20 on dist-test-slave-9fn1"
server_key: "e63897366a59f8cd7c9583d4de60f961"
server_key_iv: "130e2f56881f8d15144a3946df274582"
server_key_version: "encryptionkey@0"
I20260924 19:00:20.787057 4335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-2/wal/instance:
uuid: "89d8a7e36bbe409aa14cb4eff3c909d2"
format_stamp: "Formatted at 2026-09-24 19:00:20 on dist-test-slave-9fn1"
server_key: "e63897366a59f8cd7c9583d4de60f961"
server_key_iv: "130e2f56881f8d15144a3946df274582"
server_key_version: "encryptionkey@0"
I20260924 19:00:20.792104 4335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:00:20.795408 4350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:20.796619 4335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 19:00:20.796784 4335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "89d8a7e36bbe409aa14cb4eff3c909d2"
format_stamp: "Formatted at 2026-09-24 19:00:20 on dist-test-slave-9fn1"
server_key: "e63897366a59f8cd7c9583d4de60f961"
server_key_iv: "130e2f56881f8d15144a3946df274582"
server_key_version: "encryptionkey@0"
I20260924 19:00:20.796977 4335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:20.816305 4335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:20.819974 4335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:20.820269 4335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:20.821182 4335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:20.822455 4335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:20.822564 4335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:20.822708 4335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:20.822777 4335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:20.854969 4335 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34241
I20260924 19:00:20.855014 4462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34241 every 8 connection(s)
I20260924 19:00:20.856289 4335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:00:20.858381 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 4335
I20260924 19:00:20.858584 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276386554690-388-0/minicluster-data/ts-2/wal/instance
I20260924 19:00:20.859107 388 external_mini_cluster.cc:1428] Setting key cc12bd1c4073d2e756bfa9fef44ad34b
Sep 24 19:00:20 dist-test-slave-9fn1 krb5kdc[3962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:20.876152 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:20.858858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34389 (local address 127.0.97.62:43171)
0924 19:00:20.859025 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:20.859030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:20.860578 (+ 1548us) server_negotiation.cc:407] Connection header received
0924 19:00:20.862768 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:20.862773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:20.862863 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:20.862962 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:20.864370 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:20.865139 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:20.866748 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:20.867157 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:20.867283 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:20.870576 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:20.870581 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:20.870584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:20.870614 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:20.872949 (+ 2335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:20.873771 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:20.873776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:20.873780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:20.873911 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:20.874382 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:20.874386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:20.874390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:20.874717 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:20.874858 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:20.875577 (+ 719us) server_negotiation.cc:299] Negotiation successful
0924 19:00:20.875797 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 19:00:20.877074 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:20.859535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43171 (local address 127.0.97.3:34389)
0924 19:00:20.860419 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:20.860478 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:20.862529 (+ 2051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:20.863182 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:20.863190 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:20.863673 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:20.864209 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:20.864224 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:20.865282 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:20.865287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:20.866602 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:20.866613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:20.867381 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:20.867388 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:20.867614 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:20.868239 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:20.868263 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:20.870396 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:20.873096 (+ 2700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:20.873104 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:20.873120 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:20.873651 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:20.874031 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:20.874038 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:20.874042 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:20.874269 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:20.874833 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:20.874841 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:20.875167 (+ 326us) client_negotiation.cc:769] Sending connection context
0924 19:00:20.875492 (+ 325us) client_negotiation.cc:240] Negotiation successful
0924 19:00:20.875995 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":320,"threads_started":1}
I20260924 19:00:20.878358 4463 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43171
I20260924 19:00:20.878639 4463 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:20.879339 4463 heartbeater.cc:511] Master 127.0.97.62:43171 requested a full tablet report, sending...
I20260924 19:00:20.880645 4007 ts_manager.cc:194] Registered new tserver with Master: 89d8a7e36bbe409aa14cb4eff3c909d2 (127.0.97.3:34241)
I20260924 19:00:20.881438 4007 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34389
I20260924 19:00:20.886175 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 19:00:20 dist-test-slave-9fn1 krb5kdc[3962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260924 19:00:20.912348 4196 heartbeater.cc:503] Master 127.0.97.62:43171 was elected leader, sending a full tablet report...
Sep 24 19:00:20 dist-test-slave-9fn1 krb5kdc[3962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 19:00:20.931782 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:20.917627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54152 (local address 127.0.97.1:39055)
0924 19:00:20.918212 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:20.918219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:20.918247 (+ 28us) server_negotiation.cc:407] Connection header received
0924 19:00:20.918374 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:20.918381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:20.918594 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:20.918726 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:20.920065 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:20.921066 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:20.922351 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:20.922915 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:20.923018 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:20.926733 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:20.926745 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:20.926760 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:20.926793 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:20.929415 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:20.930052 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:20.930058 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:20.930061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:20.930182 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:20.930500 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:20.930503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:20.930505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:20.930805 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:20.931003 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:20.931243 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 19:00:20.931462 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":285,"threads_started":1}
Sep 24 19:00:20 dist-test-slave-9fn1 krb5kdc[3962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:20.949342 4198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:20.935834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36132 (local address 127.0.97.62:43171)
0924 19:00:20.936051 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:20.936056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:20.936217 (+ 161us) server_negotiation.cc:407] Connection header received
0924 19:00:20.936406 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:20.936410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:20.936467 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:20.936592 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:20.937748 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:20.938442 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:20.939736 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:20.940409 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:20.940544 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:20.944990 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:20.944995 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:20.944999 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:20.945035 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:20.947310 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:20.947865 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:20.947868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:20.947871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:20.947980 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:20.948266 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:20.948268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:20.948270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:20.948563 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:20.948728 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:20.948958 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 19:00:20.949111 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260924 19:00:20.949970 4007 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36132
I20260924 19:00:20.950910 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 4068
I20260924 19:00:20.962599 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:20.951845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54154 (local address 127.0.97.1:39055)
0924 19:00:20.952052 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:20.952057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:20.952155 (+ 98us) server_negotiation.cc:407] Connection header received
0924 19:00:20.952278 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:20.952282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:20.952360 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:20.952437 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:20.953585 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:20.954244 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:20.955411 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:20.955825 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:20.955916 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:20.958056 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:20.958061 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:20.958064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:20.958093 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:20.960531 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:20.961059 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:20.961063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:20.961065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:20.961168 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:20.961445 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:20.961448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:20.961449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:20.961709 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:20.961882 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:20.962123 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 19:00:20.962265 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 19:00:20.963150 4191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:21.044373 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 4068
I20260924 19:00:21.060249 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 4202
Sep 24 19:00:21 dist-test-slave-9fn1 krb5kdc[3962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 19:00:21.075796 4475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:21.061088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41522 (local address 127.0.97.2:36759)
0924 19:00:21.061661 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:21.061668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:21.061693 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:00:21.061751 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:21.061757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:21.061974 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:21.062148 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:21.063263 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:21.064297 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:21.065821 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:21.066371 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:21.066509 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:21.070591 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:21.070602 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:21.070615 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:21.070656 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:21.073471 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:21.074169 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:21.074176 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:21.074179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:21.074322 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:21.074595 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:21.074598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:21.074600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:21.074911 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:21.075106 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:21.075321 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 19:00:21.075501 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":261,"threads_started":1}
I20260924 19:00:21.077014 4325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:21.156870 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 4202
I20260924 19:00:21.172772 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 4335
Sep 24 19:00:21 dist-test-slave-9fn1 krb5kdc[3962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 19:00:21.187815 4477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:21.173799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44880 (local address 127.0.97.3:34241)
0924 19:00:21.174383 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:21.174390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:21.174416 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:00:21.174470 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:21.174476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:21.174676 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:21.174802 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:21.175993 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:21.177006 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:21.178302 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:21.178837 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:21.178931 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:21.182961 (+ 4030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:21.182970 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:21.182983 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:21.183012 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:21.185688 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:21.186242 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:21.186248 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:21.186250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:21.186356 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:21.186633 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:21.186636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:21.186637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:21.186942 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:21.187168 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:21.187344 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 19:00:21.187525 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":259,"threads_started":1}
I20260924 19:00:21.189024 4458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:21.269826 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 4335
I20260924 19:00:21.285415 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 3975
I20260924 19:00:21.286574 4037 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:21.388602 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 3975
2026-09-24T19:00:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3759 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:00:21 dist-test-slave-9fn1 krb5kdc[4482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:00:21 dist-test-slave-9fn1 krb5kdc[4482](info): set up 2 sockets
Sep 24 19:00:21 dist-test-slave-9fn1 krb5kdc[4482](info): commencing operation
krb5kdc: starting...
W20260924 19:00:22.473963 388 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:00:22 dist-test-slave-9fn1 krb5kdc[4482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:00:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:00:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 19:00:22.668637 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35561
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:22.977439 4495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:22.977768 4495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:22.977927 4495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:22.984774 4495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:00:22.984859 4495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:22.984903 4495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:22.985011 4495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:00:22.985078 4495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:00:22.994294 4495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:22.995481 4495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:22.996788 4495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:23.008392 4500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:23.008314 4501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:23.008368 4503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:23.009153 4495 server_base.cc:1034] running on GCE node
I20260924 19:00:23.009966 4495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:23.011368 4495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:23.012586 4495 hybrid_clock.cc:648] HybridClock initialized: now 1790276423012543 us; error 38 us; skew 500 ppm
I20260924 19:00:23.012919 4495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:23 dist-test-slave-9fn1 krb5kdc[4482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:23.016037 4495 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:00:23.017123 4495 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:23.017230 4495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:23.017603 4495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:23.020470 4495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "9418c1f23ac844b3a0245338698772a6"
format_stamp: "Formatted at 2026-09-24 19:00:23 on dist-test-slave-9fn1"
server_key: "7236ddd6655bd0ce85ed5b509f8e7362"
server_key_iv: "dce5979333b132ba99a9d22d2b47e952"
server_key_version: "encryptionkey@0"
I20260924 19:00:23.021239 4495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "9418c1f23ac844b3a0245338698772a6"
format_stamp: "Formatted at 2026-09-24 19:00:23 on dist-test-slave-9fn1"
server_key: "7236ddd6655bd0ce85ed5b509f8e7362"
server_key_iv: "dce5979333b132ba99a9d22d2b47e952"
server_key_version: "encryptionkey@0"
I20260924 19:00:23.026024 4495 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 19:00:23.029225 4509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:23.030496 4495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:00:23.030670 4495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "9418c1f23ac844b3a0245338698772a6"
format_stamp: "Formatted at 2026-09-24 19:00:23 on dist-test-slave-9fn1"
server_key: "7236ddd6655bd0ce85ed5b509f8e7362"
server_key_iv: "dce5979333b132ba99a9d22d2b47e952"
server_key_version: "encryptionkey@0"
I20260924 19:00:23.030875 4495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:23.044893 4495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:23.049009 4495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:23.049294 4495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:23.066474 4495 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35561
I20260924 19:00:23.066494 4560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35561 every 8 connection(s)
I20260924 19:00:23.067320 4495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:00:23.071046 4561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:23.075176 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 4495
I20260924 19:00:23.075405 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:00:23.076185 388 external_mini_cluster.cc:1428] Setting key 581cf7fc4f71fae4afc7717ab5a45948
I20260924 19:00:23.084728 4561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9418c1f23ac844b3a0245338698772a6: Bootstrap starting.
I20260924 19:00:23.088531 4561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9418c1f23ac844b3a0245338698772a6: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:23.089869 4561 log.cc:824] T 00000000000000000000000000000000 P 9418c1f23ac844b3a0245338698772a6: Log is configured to *not* fsync() on all Append() calls
Sep 24 19:00:23 dist-test-slave-9fn1 krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:23.092816 4561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9418c1f23ac844b3a0245338698772a6: No bootstrap required, opened a new log
I20260924 19:00:23.098677 4564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:23.078239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42728 (local address 127.0.97.62:35561)
0924 19:00:23.079026 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:23.079038 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:23.079078 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:00:23.080573 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:23.080593 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:23.080969 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:23.081292 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:23.083088 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:23.084399 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:23.086257 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:23.086924 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:23.087052 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:23.091860 (+ 4808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:23.091867 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:23.091879 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:23.091914 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:23.094832 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:23.095596 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:23.095602 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:23.095607 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:23.095760 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:23.096138 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:23.096141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:23.096143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:23.096656 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:23.096879 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:23.097252 (+ 373us) server_negotiation.cc:299] Negotiation successful
0924 19:00:23.097665 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":377,"threads_started":1}
I20260924 19:00:23.102325 4561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9418c1f23ac844b3a0245338698772a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9418c1f23ac844b3a0245338698772a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35561 } }
I20260924 19:00:23.102680 4561 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9418c1f23ac844b3a0245338698772a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:23.102780 4561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9418c1f23ac844b3a0245338698772a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9418c1f23ac844b3a0245338698772a6, State: Initialized, Role: FOLLOWER
I20260924 19:00:23.103298 4561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9418c1f23ac844b3a0245338698772a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9418c1f23ac844b3a0245338698772a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35561 } }
I20260924 19:00:23.103550 4561 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9418c1f23ac844b3a0245338698772a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:00:23.103709 4561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9418c1f23ac844b3a0245338698772a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:00:23.103888 4561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9418c1f23ac844b3a0245338698772a6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:23.106115 4561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9418c1f23ac844b3a0245338698772a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9418c1f23ac844b3a0245338698772a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35561 } }
I20260924 19:00:23.106563 4561 leader_election.cc:302] T 00000000000000000000000000000000 P 9418c1f23ac844b3a0245338698772a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9418c1f23ac844b3a0245338698772a6; no voters:
I20260924 19:00:23.107096 4561 leader_election.cc:288] T 00000000000000000000000000000000 P 9418c1f23ac844b3a0245338698772a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:00:23.107260 4566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9418c1f23ac844b3a0245338698772a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:00:23.107926 4566 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9418c1f23ac844b3a0245338698772a6 [term 1 LEADER]: Becoming Leader. State: Replica: 9418c1f23ac844b3a0245338698772a6, State: Running, Role: LEADER
I20260924 19:00:23.108594 4566 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9418c1f23ac844b3a0245338698772a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9418c1f23ac844b3a0245338698772a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35561 } }
I20260924 19:00:23.109364 4561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9418c1f23ac844b3a0245338698772a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:00:23.111593 4567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9418c1f23ac844b3a0245338698772a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9418c1f23ac844b3a0245338698772a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9418c1f23ac844b3a0245338698772a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35561 } } }
I20260924 19:00:23.112000 4567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9418c1f23ac844b3a0245338698772a6 [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:23.112401 4568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9418c1f23ac844b3a0245338698772a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9418c1f23ac844b3a0245338698772a6. Latest consensus state: current_term: 1 leader_uuid: "9418c1f23ac844b3a0245338698772a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9418c1f23ac844b3a0245338698772a6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35561 } } }
I20260924 19:00:23.112751 4568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9418c1f23ac844b3a0245338698772a6 [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:23.115146 4575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:00:23.120273 4575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:00:23.128626 4575 catalog_manager.cc:1347] Generated new cluster ID: 437d144f1f5c400c9eae68dc36ea72c0
I20260924 19:00:23.128759 4575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:00:23.146127 4575 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:00:23.147372 4575 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:00:23.152184 4575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9418c1f23ac844b3a0245338698772a6: Generated new TSK 0
I20260924 19:00:23.152951 4575 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 19:00:23.204568 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35561
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:23.525305 4587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:23.525578 4587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:23.525674 4587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:23.532639 4587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:23.532748 4587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:23.532863 4587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:00:23.542299 4587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:23.543401 4587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:23.544530 4587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:23.556702 4593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:23.556746 4592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:23.556665 4595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:23.557480 4587 server_base.cc:1034] running on GCE node
I20260924 19:00:23.558072 4587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:23.558943 4587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:23.560168 4587 hybrid_clock.cc:648] HybridClock initialized: now 1790276423560117 us; error 47 us; skew 500 ppm
I20260924 19:00:23.560442 4587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:23 dist-test-slave-9fn1 krb5kdc[4482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:23.563339 4587 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 19:00:23.564441 4587 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:23.564560 4587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:23.564842 4587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:23.567559 4587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-0/data/instance:
uuid: "8acbf17962684e44b5064675570b485a"
format_stamp: "Formatted at 2026-09-24 19:00:23 on dist-test-slave-9fn1"
server_key: "6dc7fd221ef250226dc88159e7739236"
server_key_iv: "3cef1cf5db46f47300ae116b652c0bf4"
server_key_version: "encryptionkey@0"
I20260924 19:00:23.568333 4587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8acbf17962684e44b5064675570b485a"
format_stamp: "Formatted at 2026-09-24 19:00:23 on dist-test-slave-9fn1"
server_key: "6dc7fd221ef250226dc88159e7739236"
server_key_iv: "3cef1cf5db46f47300ae116b652c0bf4"
server_key_version: "encryptionkey@0"
I20260924 19:00:23.573031 4587 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260924 19:00:23.576095 4601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:23.577217 4587 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 19:00:23.577382 4587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "8acbf17962684e44b5064675570b485a"
format_stamp: "Formatted at 2026-09-24 19:00:23 on dist-test-slave-9fn1"
server_key: "6dc7fd221ef250226dc88159e7739236"
server_key_iv: "3cef1cf5db46f47300ae116b652c0bf4"
server_key_version: "encryptionkey@0"
I20260924 19:00:23.577649 4587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:23.594283 4587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:23.597824 4587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:23.598078 4587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:23.598897 4587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:23.600144 4587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:23.600260 4587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:23.600370 4587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:23.600456 4587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:23.631740 4587 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37205
I20260924 19:00:23.631820 4713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37205 every 8 connection(s)
I20260924 19:00:23.632565 4587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:00:23.642805 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 4587
I20260924 19:00:23.643029 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-0/wal/instance
I20260924 19:00:23.643687 388 external_mini_cluster.cc:1428] Setting key 47edd70834d87a0847e2ab73cd59b81c
Sep 24 19:00:23 dist-test-slave-9fn1 krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:23.651188 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:23.634926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43825 (local address 127.0.97.62:35561)
0924 19:00:23.635381 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:23.635387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:23.636343 (+ 956us) server_negotiation.cc:407] Connection header received
0924 19:00:23.638120 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:23.638124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:23.638200 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:23.638297 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:23.639651 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:23.640365 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:23.641883 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:23.642262 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:23.642354 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:23.645821 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:23.645826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:23.645829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:23.645857 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:23.648227 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:23.648927 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:23.648936 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:23.648940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:23.649046 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:23.649456 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:23.649460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:23.649463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:23.649769 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:23.649921 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:23.650567 (+ 646us) server_negotiation.cc:299] Negotiation successful
0924 19:00:23.650791 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":243,"threads_started":1}
I20260924 19:00:23.652145 4717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:23.635524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35561 (local address 127.0.97.1:43825)
0924 19:00:23.636192 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:23.636228 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:23.637886 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:23.638478 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:23.638485 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:23.638982 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:23.639511 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:23.639531 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:23.640479 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:23.640483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:23.641774 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:23.641785 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:23.642434 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:23.642438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:23.642623 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:23.643339 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:23.643368 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:23.645612 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:23.648365 (+ 2753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:23.648374 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:23.648390 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:23.648794 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:23.649170 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:23.649174 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:23.649178 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:23.649346 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:23.649891 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:23.649899 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:23.650217 (+ 318us) client_negotiation.cc:769] Sending connection context
0924 19:00:23.650482 (+ 265us) client_negotiation.cc:240] Negotiation successful
0924 19:00:23.651022 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":275,"threads_started":1}
I20260924 19:00:23.653647 4714 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35561
I20260924 19:00:23.653950 4714 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:23.654675 4714 heartbeater.cc:511] Master 127.0.97.62:35561 requested a full tablet report, sending...
I20260924 19:00:23.656636 4526 ts_manager.cc:194] Registered new tserver with Master: 8acbf17962684e44b5064675570b485a (127.0.97.1:37205)
I20260924 19:00:23.658144 4526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43825
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 19:00:23.682219 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35561
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:23.993906 4720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:23.994220 4720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:23.994396 4720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:24.001485 4720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:24.001667 4720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:24.001858 4720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:00:24.011675 4720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:24.013094 4720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:24.014429 4720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:24.026556 4726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:24.026592 4728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:24.026783 4725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:24.027853 4720 server_base.cc:1034] running on GCE node
I20260924 19:00:24.028439 4720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:24.029318 4720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:24.030517 4720 hybrid_clock.cc:648] HybridClock initialized: now 1790276424030449 us; error 57 us; skew 500 ppm
I20260924 19:00:24.030831 4720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:24 dist-test-slave-9fn1 krb5kdc[4482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:24.034011 4720 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 19:00:24.035247 4720 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:24.035357 4720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:24.035677 4720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:24.038642 4720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-1/data/instance:
uuid: "1d0693a9a55845948a2c382d60e351c2"
format_stamp: "Formatted at 2026-09-24 19:00:24 on dist-test-slave-9fn1"
server_key: "5164e5aaea6ee0e3856614d6e0def18c"
server_key_iv: "ab8a77f89f20f2fa1ba7587b57a4d7fb"
server_key_version: "encryptionkey@0"
I20260924 19:00:24.039474 4720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1d0693a9a55845948a2c382d60e351c2"
format_stamp: "Formatted at 2026-09-24 19:00:24 on dist-test-slave-9fn1"
server_key: "5164e5aaea6ee0e3856614d6e0def18c"
server_key_iv: "ab8a77f89f20f2fa1ba7587b57a4d7fb"
server_key_version: "encryptionkey@0"
I20260924 19:00:24.044546 4720 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:00:24.047910 4734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:24.049208 4720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:00:24.049383 4720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "1d0693a9a55845948a2c382d60e351c2"
format_stamp: "Formatted at 2026-09-24 19:00:24 on dist-test-slave-9fn1"
server_key: "5164e5aaea6ee0e3856614d6e0def18c"
server_key_iv: "ab8a77f89f20f2fa1ba7587b57a4d7fb"
server_key_version: "encryptionkey@0"
I20260924 19:00:24.049670 4720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:24.073132 4720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:24.077462 4720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:24.077760 4720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:24.078831 4720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:24.080163 4720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:24.080284 4720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:24.080404 4720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:24.080498 4720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:24.112164 4720 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40203
I20260924 19:00:24.112247 4846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40203 every 8 connection(s)
I20260924 19:00:24.113030 4720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:00:24.117826 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 4720
I20260924 19:00:24.118019 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-1/wal/instance
I20260924 19:00:24.118526 388 external_mini_cluster.cc:1428] Setting key 7b4ecf80c044cac9af4c3efccaf4dba6
Sep 24 19:00:24 dist-test-slave-9fn1 krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:24.135577 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:24.115462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39967 (local address 127.0.97.62:35561)
0924 19:00:24.115634 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:24.115639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:24.116762 (+ 1123us) server_negotiation.cc:407] Connection header received
0924 19:00:24.119213 (+ 2451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:24.119218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:24.119308 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:24.119431 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:24.121433 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.122657 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.124184 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.124923 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.125066 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:24.128758 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:24.128767 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:24.128774 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:24.128818 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:24.132349 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:24.133032 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:24.133040 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:24.133046 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:24.133212 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:24.133662 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:24.133670 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:24.133676 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:24.134050 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:24.134199 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:24.134870 (+ 671us) server_negotiation.cc:299] Negotiation successful
0924 19:00:24.135127 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 19:00:24.136425 4849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:24.115933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35561 (local address 127.0.97.2:39967)
0924 19:00:24.116601 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:24.116645 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:24.118848 (+ 2203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:24.119799 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:24.119810 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:24.120538 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:24.121272 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:24.121290 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.122798 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.122803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:24.124043 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:24.124054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.125112 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.125116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:24.125288 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:24.125979 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:24.125999 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:24.128545 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:24.132503 (+ 3958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:24.132509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:24.132522 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:24.132916 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:24.133326 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:24.133329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:24.133331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:24.133562 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:24.134178 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:24.134184 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:24.134509 (+ 325us) client_negotiation.cc:769] Sending connection context
0924 19:00:24.134744 (+ 235us) client_negotiation.cc:240] Negotiation successful
0924 19:00:24.135246 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":279,"threads_started":1}
I20260924 19:00:24.137894 4847 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35561
I20260924 19:00:24.138247 4847 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:24.139045 4847 heartbeater.cc:511] Master 127.0.97.62:35561 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 19:00:24.140590 4526 ts_manager.cc:194] Registered new tserver with Master: 1d0693a9a55845948a2c382d60e351c2 (127.0.97.2:40203)
I20260924 19:00:24.141407 4526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39967
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 19:00:24.163947 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35561
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:24.473747 4852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:24.474015 4852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:24.474108 4852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:24.481271 4852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:24.481385 4852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:24.481513 4852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:00:24.491115 4852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35561
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:24.492321 4852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:24.493602 4852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:24.505973 4857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:24.505937 4858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:24.506006 4860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:24.506696 4852 server_base.cc:1034] running on GCE node
I20260924 19:00:24.507308 4852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:24.508177 4852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:24.509415 4852 hybrid_clock.cc:648] HybridClock initialized: now 1790276424509372 us; error 49 us; skew 500 ppm
I20260924 19:00:24.509765 4852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:24 dist-test-slave-9fn1 krb5kdc[4482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:24.518604 4852 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 19:00:24.519811 4852 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:24.519935 4852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:24.520255 4852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:24.523371 4852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-2/data/instance:
uuid: "63060188142e43de9db90e56c39556bc"
format_stamp: "Formatted at 2026-09-24 19:00:24 on dist-test-slave-9fn1"
server_key: "684cded62b77567d81e4863cee1b9291"
server_key_iv: "0a60f750677582b70931776aff72f248"
server_key_version: "encryptionkey@0"
I20260924 19:00:24.524135 4852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-2/wal/instance:
uuid: "63060188142e43de9db90e56c39556bc"
format_stamp: "Formatted at 2026-09-24 19:00:24 on dist-test-slave-9fn1"
server_key: "684cded62b77567d81e4863cee1b9291"
server_key_iv: "0a60f750677582b70931776aff72f248"
server_key_version: "encryptionkey@0"
I20260924 19:00:24.528887 4852 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 19:00:24.532073 4866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:24.533254 4852 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 19:00:24.533411 4852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "63060188142e43de9db90e56c39556bc"
format_stamp: "Formatted at 2026-09-24 19:00:24 on dist-test-slave-9fn1"
server_key: "684cded62b77567d81e4863cee1b9291"
server_key_iv: "0a60f750677582b70931776aff72f248"
server_key_version: "encryptionkey@0"
I20260924 19:00:24.533667 4852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:24.564702 4852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:24.568560 4852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:24.568815 4852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:24.569764 4852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:24.571004 4852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:24.571130 4852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:24.571241 4852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:24.571326 4852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:24.607513 4852 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35609
I20260924 19:00:24.607558 4978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35609 every 8 connection(s)
I20260924 19:00:24.608335 4852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:00:24.611023 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 4852
I20260924 19:00:24.611215 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-2/wal/instance
I20260924 19:00:24.611554 388 external_mini_cluster.cc:1428] Setting key 4266f4fc015d7c57abceac16c431b8bb
Sep 24 19:00:24 dist-test-slave-9fn1 krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:24.627790 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:24.610902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52133 (local address 127.0.97.62:35561)
0924 19:00:24.611116 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:24.611124 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:24.612357 (+ 1233us) server_negotiation.cc:407] Connection header received
0924 19:00:24.614713 (+ 2356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:24.614716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:24.614807 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:24.614925 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:24.616366 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.617077 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.618572 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.618985 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.619077 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:24.622267 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:24.622272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:24.622275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:24.622324 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:24.624766 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:24.625486 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:24.625493 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:24.625517 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:24.625652 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:24.626166 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:24.626170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:24.626173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:24.626509 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:24.626644 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:24.627421 (+ 777us) server_negotiation.cc:299] Negotiation successful
0924 19:00:24.627567 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 19:00:24.628765 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:24.611393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35561 (local address 127.0.97.3:52133)
0924 19:00:24.612206 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:24.612251 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:24.614448 (+ 2197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:24.615192 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:24.615199 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:24.615680 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:24.616226 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:24.616242 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.617186 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.617190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:24.618462 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:24.618473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.619161 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.619166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:24.619331 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:24.619911 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:24.619932 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:24.622047 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:24.624872 (+ 2825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:24.624878 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:24.624890 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:24.625344 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:24.625776 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:24.625780 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:24.625783 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:24.626021 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:24.626647 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:24.626653 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:24.627066 (+ 413us) client_negotiation.cc:769] Sending connection context
0924 19:00:24.627310 (+ 244us) client_negotiation.cc:240] Negotiation successful
0924 19:00:24.627667 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":412,"threads_started":1}
I20260924 19:00:24.630108 4979 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35561
I20260924 19:00:24.630383 4979 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:24.630964 4979 heartbeater.cc:511] Master 127.0.97.62:35561 requested a full tablet report, sending...
I20260924 19:00:24.632427 4526 ts_manager.cc:194] Registered new tserver with Master: 63060188142e43de9db90e56c39556bc (127.0.97.3:35609)
I20260924 19:00:24.633214 4526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52133
I20260924 19:00:24.639117 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:00:24.654893 4716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:24.643347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43350 (local address 127.0.97.62:35561)
0924 19:00:24.643588 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:24.643594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:24.643864 (+ 270us) server_negotiation.cc:407] Connection header received
0924 19:00:24.644003 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:24.644007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:24.644067 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:24.644148 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:24.645557 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.646234 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.647404 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.647811 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.647900 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:24.650134 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:24.650138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:24.650142 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:24.650168 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:24.652440 (+ 2272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:24.653070 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:24.653073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:24.653076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:24.653201 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:24.653540 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:24.653556 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:24.653558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:24.653920 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:24.654094 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:24.654397 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 19:00:24.654568 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 19:00:24.660786 4714 heartbeater.cc:503] Master 127.0.97.62:35561 was elected leader, sending a full tablet report...
I20260924 19:00:24.658742 4526 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:00:24.665795 4526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:00:24.701537 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:24.678105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41632 (local address 127.0.97.2:40203)
0924 19:00:24.689236 (+ 11131us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:24.689249 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:24.689291 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:00:24.689433 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:24.689444 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:24.689808 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:24.691208 (+ 1400us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:24.691509 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.693561 (+ 2052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.698529 (+ 4968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.699955 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.700087 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:24.700296 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:24.700959 (+ 663us) server_negotiation.cc:299] Negotiation successful
0924 19:00:24.701171 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10895,"thread_start_us":11905,"threads_started":1}
I20260924 19:00:24.701601 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:24.677546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40203 (local address 127.0.0.1:41632)
0924 19:00:24.678383 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:24.678437 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:24.678657 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:24.690387 (+ 11730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:24.690398 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:24.690445 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:24.690944 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:24.690960 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.694542 (+ 3582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.694547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:24.698364 (+ 3817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:24.698376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.700433 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.700439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:24.700668 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:24.700747 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 19:00:24.700903 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 19:00:24.701108 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":5,"thread_start_us":744,"threads_started":1}
I20260924 19:00:24.704317 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:24.677689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37205 (local address 127.0.0.1:41386)
0924 19:00:24.694186 (+ 16497us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:24.694212 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:24.694343 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:24.696164 (+ 1821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:24.696171 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:24.696205 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:24.697991 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:24.698006 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.699890 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.699895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:24.702061 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:24.702078 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.703561 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.703567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:24.703738 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:24.703771 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 19:00:24.703859 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 19:00:24.703998 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16295,"thread_start_us":16148,"threads_started":1}
I20260924 19:00:24.704432 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:24.677992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41386 (local address 127.0.97.1:37205)
0924 19:00:24.695406 (+ 17414us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:24.695418 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:24.695455 (+ 37us) server_negotiation.cc:407] Connection header received
0924 19:00:24.695567 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:24.695579 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:24.695877 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:24.696410 (+ 533us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:24.698218 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.699710 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.702280 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.703346 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.703489 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:24.703723 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:24.703935 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 19:00:24.704106 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17212,"thread_start_us":17056,"threads_started":1}
I20260924 19:00:24.706061 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:24.679642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35609 (local address 127.0.0.1:43614)
0924 19:00:24.697929 (+ 18287us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:24.697951 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:24.698080 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:24.698654 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:24.698658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:24.698682 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:24.699061 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:24.699069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.700674 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.700681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:24.703433 (+ 2752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:24.703449 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.705276 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.705282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:24.705438 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:24.705468 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 19:00:24.705597 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 19:00:24.705712 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18138,"mutex_wait_us":27,"thread_start_us":17987,"threads_started":1}
I20260924 19:00:24.706056 4782 tablet_service.cc:1467] Processing CreateTablet for tablet 5af697387ec049778e40503779111de1 (DEFAULT_TABLE table=test-table [id=b89a0983035541a2928f28136904e642]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:24.706750 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:24.680235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43614 (local address 127.0.97.3:35609)
0924 19:00:24.697127 (+ 16892us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:24.697132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:24.698069 (+ 937us) server_negotiation.cc:407] Connection header received
0924 19:00:24.698243 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:24.698252 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:24.698480 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:24.698672 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:24.699207 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.700531 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.703630 (+ 3099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.705085 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.705205 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:24.705425 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:24.706288 (+ 863us) server_negotiation.cc:299] Negotiation successful
0924 19:00:24.706435 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16738,"thread_start_us":16593,"threads_started":1}
I20260924 19:00:24.707422 4782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5af697387ec049778e40503779111de1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:24.707628 4649 tablet_service.cc:1467] Processing CreateTablet for tablet 5af697387ec049778e40503779111de1 (DEFAULT_TABLE table=test-table [id=b89a0983035541a2928f28136904e642]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:24.708688 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5af697387ec049778e40503779111de1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:24.714087 4914 tablet_service.cc:1467] Processing CreateTablet for tablet 5af697387ec049778e40503779111de1 (DEFAULT_TABLE table=test-table [id=b89a0983035541a2928f28136904e642]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:24.715523 4914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5af697387ec049778e40503779111de1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:24.719164 4997 tablet_bootstrap.cc:492] T 5af697387ec049778e40503779111de1 P 1d0693a9a55845948a2c382d60e351c2: Bootstrap starting.
I20260924 19:00:24.722245 4997 tablet_bootstrap.cc:654] T 5af697387ec049778e40503779111de1 P 1d0693a9a55845948a2c382d60e351c2: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:24.723060 4998 tablet_bootstrap.cc:492] T 5af697387ec049778e40503779111de1 P 8acbf17962684e44b5064675570b485a: Bootstrap starting.
I20260924 19:00:24.723567 4997 log.cc:824] T 5af697387ec049778e40503779111de1 P 1d0693a9a55845948a2c382d60e351c2: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:24.726428 4998 tablet_bootstrap.cc:654] T 5af697387ec049778e40503779111de1 P 8acbf17962684e44b5064675570b485a: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:24.726732 4997 tablet_bootstrap.cc:492] T 5af697387ec049778e40503779111de1 P 1d0693a9a55845948a2c382d60e351c2: No bootstrap required, opened a new log
I20260924 19:00:24.727021 4997 ts_tablet_manager.cc:1397] T 5af697387ec049778e40503779111de1 P 1d0693a9a55845948a2c382d60e351c2: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260924 19:00:24.727656 4998 log.cc:824] T 5af697387ec049778e40503779111de1 P 8acbf17962684e44b5064675570b485a: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:24.728487 5000 tablet_bootstrap.cc:492] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc: Bootstrap starting.
I20260924 19:00:24.731366 5000 tablet_bootstrap.cc:654] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:24.731779 4998 tablet_bootstrap.cc:492] T 5af697387ec049778e40503779111de1 P 8acbf17962684e44b5064675570b485a: No bootstrap required, opened a new log
I20260924 19:00:24.732055 4998 ts_tablet_manager.cc:1397] T 5af697387ec049778e40503779111de1 P 8acbf17962684e44b5064675570b485a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 19:00:24.732399 5000 log.cc:824] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:24.735011 5000 tablet_bootstrap.cc:492] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc: No bootstrap required, opened a new log
I20260924 19:00:24.735246 5000 ts_tablet_manager.cc:1397] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 19:00:24.737870 4997 raft_consensus.cc:348] T 5af697387ec049778e40503779111de1 P 1d0693a9a55845948a2c382d60e351c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0693a9a55845948a2c382d60e351c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40203 } } peers { permanent_uuid: "8acbf17962684e44b5064675570b485a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37205 } } peers { permanent_uuid: "63060188142e43de9db90e56c39556bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } }
I20260924 19:00:24.738343 4997 raft_consensus.cc:374] T 5af697387ec049778e40503779111de1 P 1d0693a9a55845948a2c382d60e351c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:24.738461 4997 raft_consensus.cc:729] T 5af697387ec049778e40503779111de1 P 1d0693a9a55845948a2c382d60e351c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0693a9a55845948a2c382d60e351c2, State: Initialized, Role: FOLLOWER
I20260924 19:00:24.739107 4997 consensus_queue.cc:260] T 5af697387ec049778e40503779111de1 P 1d0693a9a55845948a2c382d60e351c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0693a9a55845948a2c382d60e351c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40203 } } peers { permanent_uuid: "8acbf17962684e44b5064675570b485a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37205 } } peers { permanent_uuid: "63060188142e43de9db90e56c39556bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } }
I20260924 19:00:24.740219 4847 heartbeater.cc:503] Master 127.0.97.62:35561 was elected leader, sending a full tablet report...
I20260924 19:00:24.740901 4997 ts_tablet_manager.cc:1428] T 5af697387ec049778e40503779111de1 P 1d0693a9a55845948a2c382d60e351c2: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260924 19:00:24.742781 4998 raft_consensus.cc:348] T 5af697387ec049778e40503779111de1 P 8acbf17962684e44b5064675570b485a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0693a9a55845948a2c382d60e351c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40203 } } peers { permanent_uuid: "8acbf17962684e44b5064675570b485a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37205 } } peers { permanent_uuid: "63060188142e43de9db90e56c39556bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } }
I20260924 19:00:24.743115 4998 raft_consensus.cc:374] T 5af697387ec049778e40503779111de1 P 8acbf17962684e44b5064675570b485a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:24.743234 4998 raft_consensus.cc:729] T 5af697387ec049778e40503779111de1 P 8acbf17962684e44b5064675570b485a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8acbf17962684e44b5064675570b485a, State: Initialized, Role: FOLLOWER
I20260924 19:00:24.743752 4998 consensus_queue.cc:260] T 5af697387ec049778e40503779111de1 P 8acbf17962684e44b5064675570b485a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0693a9a55845948a2c382d60e351c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40203 } } peers { permanent_uuid: "8acbf17962684e44b5064675570b485a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37205 } } peers { permanent_uuid: "63060188142e43de9db90e56c39556bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } }
I20260924 19:00:24.744061 5000 raft_consensus.cc:348] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0693a9a55845948a2c382d60e351c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40203 } } peers { permanent_uuid: "8acbf17962684e44b5064675570b485a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37205 } } peers { permanent_uuid: "63060188142e43de9db90e56c39556bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } }
I20260924 19:00:24.744504 5000 raft_consensus.cc:374] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:24.744612 5000 raft_consensus.cc:729] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63060188142e43de9db90e56c39556bc, State: Initialized, Role: FOLLOWER
I20260924 19:00:24.745128 5000 consensus_queue.cc:260] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0693a9a55845948a2c382d60e351c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40203 } } peers { permanent_uuid: "8acbf17962684e44b5064675570b485a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37205 } } peers { permanent_uuid: "63060188142e43de9db90e56c39556bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } }
I20260924 19:00:24.745437 4998 ts_tablet_manager.cc:1428] T 5af697387ec049778e40503779111de1 P 8acbf17962684e44b5064675570b485a: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260924 19:00:24.746681 4979 heartbeater.cc:503] Master 127.0.97.62:35561 was elected leader, sending a full tablet report...
I20260924 19:00:24.747328 5000 ts_tablet_manager.cc:1428] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
W20260924 19:00:24.859897 4980 tablet.cc:2369] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:00:24.864830 4848 tablet.cc:2369] T 5af697387ec049778e40503779111de1 P 1d0693a9a55845948a2c382d60e351c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:00:24.884055 5005 raft_consensus.cc:482] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:00:24.884357 5005 raft_consensus.cc:504] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0693a9a55845948a2c382d60e351c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40203 } } peers { permanent_uuid: "8acbf17962684e44b5064675570b485a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37205 } } peers { permanent_uuid: "63060188142e43de9db90e56c39556bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } }
W20260924 19:00:24.884644 4715 tablet.cc:2369] T 5af697387ec049778e40503779111de1 P 8acbf17962684e44b5064675570b485a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:00:24.886138 5005 leader_election.cc:288] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0693a9a55845948a2c382d60e351c2 (127.0.97.2:40203), 8acbf17962684e44b5064675570b485a (127.0.97.1:37205)
I20260924 19:00:24.891592 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:24.886521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43885 (local address 127.0.97.2:40203)
0924 19:00:24.886774 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:24.886783 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:24.886949 (+ 166us) server_negotiation.cc:407] Connection header received
0924 19:00:24.887154 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:24.887164 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:24.887221 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:24.887313 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:24.887665 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.888372 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.889989 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.890727 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.890869 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:24.890960 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:24.891208 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 19:00:24.891326 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260924 19:00:24.891600 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:24.886432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40203 (local address 127.0.97.3:43885)
0924 19:00:24.886840 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:24.886872 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:24.887009 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:24.887332 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:24.887336 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:24.887366 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:24.887544 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:24.887553 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.888499 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.888504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:24.889848 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:24.889865 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.890886 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.890892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:24.891065 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:24.891088 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 19:00:24.891185 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 19:00:24.891311 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266}
I20260924 19:00:24.892341 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:24.886709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56277 (local address 127.0.97.1:37205)
0924 19:00:24.886893 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:24.886901 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:24.887473 (+ 572us) server_negotiation.cc:407] Connection header received
0924 19:00:24.887664 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:24.887668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:24.887722 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:24.887814 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:24.888284 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.888977 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.890656 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.891381 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.891537 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:24.891633 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:24.891984 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 19:00:24.892111 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 19:00:24.892390 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:24.886602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37205 (local address 127.0.97.3:56277)
0924 19:00:24.887360 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:24.887374 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:24.887521 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:24.887826 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:24.887830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:24.887852 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:24.888142 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:24.888150 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.889121 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.889127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:24.890496 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:24.890507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.891597 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.891602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:24.891773 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:24.891802 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 19:00:24.891955 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 19:00:24.892073 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":440,"threads_started":1}
I20260924 19:00:24.892580 4802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af697387ec049778e40503779111de1" candidate_uuid: "63060188142e43de9db90e56c39556bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0693a9a55845948a2c382d60e351c2" is_pre_election: true
I20260924 19:00:24.893181 4802 raft_consensus.cc:2454] T 5af697387ec049778e40503779111de1 P 1d0693a9a55845948a2c382d60e351c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63060188142e43de9db90e56c39556bc in term 0.
I20260924 19:00:24.893415 4669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af697387ec049778e40503779111de1" candidate_uuid: "63060188142e43de9db90e56c39556bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8acbf17962684e44b5064675570b485a" is_pre_election: true
I20260924 19:00:24.893944 4669 raft_consensus.cc:2454] T 5af697387ec049778e40503779111de1 P 8acbf17962684e44b5064675570b485a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63060188142e43de9db90e56c39556bc in term 0.
I20260924 19:00:24.893918 4869 leader_election.cc:302] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0693a9a55845948a2c382d60e351c2, 63060188142e43de9db90e56c39556bc; no voters:
I20260924 19:00:24.894326 5005 raft_consensus.cc:2781] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:00:24.894510 5005 raft_consensus.cc:482] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:00:24.894647 5005 raft_consensus.cc:3037] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:24.896768 5005 raft_consensus.cc:504] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0693a9a55845948a2c382d60e351c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40203 } } peers { permanent_uuid: "8acbf17962684e44b5064675570b485a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37205 } } peers { permanent_uuid: "63060188142e43de9db90e56c39556bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } }
I20260924 19:00:24.897368 5005 leader_election.cc:288] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc [CANDIDATE]: Term 1 election: Requested vote from peers 1d0693a9a55845948a2c382d60e351c2 (127.0.97.2:40203), 8acbf17962684e44b5064675570b485a (127.0.97.1:37205)
I20260924 19:00:24.897866 4802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af697387ec049778e40503779111de1" candidate_uuid: "63060188142e43de9db90e56c39556bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0693a9a55845948a2c382d60e351c2"
I20260924 19:00:24.897866 4669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5af697387ec049778e40503779111de1" candidate_uuid: "63060188142e43de9db90e56c39556bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8acbf17962684e44b5064675570b485a"
I20260924 19:00:24.898072 4669 raft_consensus.cc:3037] T 5af697387ec049778e40503779111de1 P 8acbf17962684e44b5064675570b485a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:24.898082 4802 raft_consensus.cc:3037] T 5af697387ec049778e40503779111de1 P 1d0693a9a55845948a2c382d60e351c2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:24.901189 4802 raft_consensus.cc:2454] T 5af697387ec049778e40503779111de1 P 1d0693a9a55845948a2c382d60e351c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63060188142e43de9db90e56c39556bc in term 1.
I20260924 19:00:24.901189 4669 raft_consensus.cc:2454] T 5af697387ec049778e40503779111de1 P 8acbf17962684e44b5064675570b485a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63060188142e43de9db90e56c39556bc in term 1.
I20260924 19:00:24.901618 4869 leader_election.cc:302] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0693a9a55845948a2c382d60e351c2, 63060188142e43de9db90e56c39556bc; no voters:
I20260924 19:00:24.901983 5005 raft_consensus.cc:2781] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:00:24.902412 5005 raft_consensus.cc:686] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc [term 1 LEADER]: Becoming Leader. State: Replica: 63060188142e43de9db90e56c39556bc, State: Running, Role: LEADER
I20260924 19:00:24.902887 5005 consensus_queue.cc:237] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0693a9a55845948a2c382d60e351c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40203 } } peers { permanent_uuid: "8acbf17962684e44b5064675570b485a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37205 } } peers { permanent_uuid: "63060188142e43de9db90e56c39556bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } }
I20260924 19:00:24.907414 4526 catalog_manager.cc:5495] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 63060188142e43de9db90e56c39556bc (127.0.97.3). New cstate: current_term: 1 leader_uuid: "63060188142e43de9db90e56c39556bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0693a9a55845948a2c382d60e351c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8acbf17962684e44b5064675570b485a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63060188142e43de9db90e56c39556bc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35609 } health_report { overall_health: HEALTHY } } }
I20260924 19:00:24.925261 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:24.920166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43628 (local address 127.0.97.3:35609)
0924 19:00:24.920369 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:24.920374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:24.920392 (+ 18us) server_negotiation.cc:407] Connection header received
0924 19:00:24.920550 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:24.920553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:24.920626 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:24.920781 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:00:24.921154 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.921918 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.923237 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.923627 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.923754 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:24.924236 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:00:24.924717 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:00:24.924868 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:24.924925 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 19:00:24.925003 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:00:24.932875 4802 raft_consensus.cc:1261] T 5af697387ec049778e40503779111de1 P 1d0693a9a55845948a2c382d60e351c2 [term 1 FOLLOWER]: Refusing update from remote peer 63060188142e43de9db90e56c39556bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:00:24.932907 4669 raft_consensus.cc:1261] T 5af697387ec049778e40503779111de1 P 8acbf17962684e44b5064675570b485a [term 1 FOLLOWER]: Refusing update from remote peer 63060188142e43de9db90e56c39556bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:00:24.933810 5007 consensus_queue.cc:1035] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8acbf17962684e44b5064675570b485a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:00:24.934158 5005 consensus_queue.cc:1035] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0693a9a55845948a2c382d60e351c2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:00:24.952685 5012 mvcc.cc:205] Tried to move back new op lower bound from 7332972236516491264 to 7332972236405755904. Current Snapshot: MvccSnapshot[applied={T|T < 7332972236516491264}]
I20260924 19:00:24.964897 4526 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:00:24.965173 4526 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:00:24.968456 4526 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9418c1f23ac844b3a0245338698772a6: Sending DeleteTablet for 3 replicas of tablet 5af697387ec049778e40503779111de1
I20260924 19:00:24.969439 4782 tablet_service.cc:1514] Processing DeleteTablet for tablet 5af697387ec049778e40503779111de1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41632
I20260924 19:00:24.969548 4914 tablet_service.cc:1514] Processing DeleteTablet for tablet 5af697387ec049778e40503779111de1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43614
I20260924 19:00:24.969604 4649 tablet_service.cc:1514] Processing DeleteTablet for tablet 5af697387ec049778e40503779111de1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:24 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41386
I20260924 19:00:24.972985 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 4587
I20260924 19:00:24.981894 5021 tablet_replica.cc:330] T 5af697387ec049778e40503779111de1 P 8acbf17962684e44b5064675570b485a: stopping tablet replica
I20260924 19:00:24.982347 5021 raft_consensus.cc:2229] T 5af697387ec049778e40503779111de1 P 8acbf17962684e44b5064675570b485a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:00:24.982831 5021 raft_consensus.cc:2258] T 5af697387ec049778e40503779111de1 P 8acbf17962684e44b5064675570b485a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:24.984306 5020 tablet_replica.cc:330] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc: stopping tablet replica
I20260924 19:00:24.984679 5020 raft_consensus.cc:2229] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:00:24.985249 5020 raft_consensus.cc:2258] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:24.985620 5021 ts_tablet_manager.cc:1905] T 5af697387ec049778e40503779111de1 P 8acbf17962684e44b5064675570b485a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:24.985966 5019 tablet_replica.cc:330] T 5af697387ec049778e40503779111de1 P 1d0693a9a55845948a2c382d60e351c2: stopping tablet replica
I20260924 19:00:24.986330 5019 raft_consensus.cc:2229] T 5af697387ec049778e40503779111de1 P 1d0693a9a55845948a2c382d60e351c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:00:24.986817 5019 raft_consensus.cc:2258] T 5af697387ec049778e40503779111de1 P 1d0693a9a55845948a2c382d60e351c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:24.987358 5020 ts_tablet_manager.cc:1905] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:24.994078 5021 ts_tablet_manager.cc:1918] T 5af697387ec049778e40503779111de1 P 8acbf17962684e44b5064675570b485a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:00:24.994295 5021 log.cc:1196] T 5af697387ec049778e40503779111de1 P 8acbf17962684e44b5064675570b485a: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-0/wal/wals/5af697387ec049778e40503779111de1
I20260924 19:00:24.994480 5020 ts_tablet_manager.cc:1918] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:00:24.994710 5020 log.cc:1196] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-2/wal/wals/5af697387ec049778e40503779111de1
I20260924 19:00:24.994774 5019 ts_tablet_manager.cc:1905] T 5af697387ec049778e40503779111de1 P 1d0693a9a55845948a2c382d60e351c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:24.994999 5021 ts_tablet_manager.cc:1939] T 5af697387ec049778e40503779111de1 P 8acbf17962684e44b5064675570b485a: Deleting consensus metadata
I20260924 19:00:24.995266 5020 ts_tablet_manager.cc:1939] T 5af697387ec049778e40503779111de1 P 63060188142e43de9db90e56c39556bc: Deleting consensus metadata
Sep 24 19:00:24 dist-test-slave-9fn1 krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 19:00:24.996976 4511 catalog_manager.cc:4841] TS 8acbf17962684e44b5064675570b485a (127.0.97.1:37205): tablet 5af697387ec049778e40503779111de1 (table test-table [id=b89a0983035541a2928f28136904e642]) successfully deleted
I20260924 19:00:24.997148 4513 catalog_manager.cc:4841] TS 63060188142e43de9db90e56c39556bc (127.0.97.3:35609): tablet 5af697387ec049778e40503779111de1 (table test-table [id=b89a0983035541a2928f28136904e642]) successfully deleted
I20260924 19:00:25.003731 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:24.974838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41400 (local address 127.0.97.1:37205)
0924 19:00:24.975069 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:24.975080 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:24.983699 (+ 8619us) server_negotiation.cc:407] Connection header received
0924 19:00:24.983896 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:24.983904 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:24.984011 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:24.984128 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:24.987116 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.988215 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.990083 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:24.990704 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:24.990795 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:24.998025 (+ 7230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:24.998036 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:24.998049 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:24.998081 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:25.000854 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:25.001647 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:25.001654 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:25.001656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:25.001792 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:25.002174 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:25.002178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:25.002180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:25.002511 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:25.002815 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:25.003094 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 19:00:25.003310 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 19:00:25.004087 5019 ts_tablet_manager.cc:1918] T 5af697387ec049778e40503779111de1 P 1d0693a9a55845948a2c382d60e351c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:00:25.004340 5019 log.cc:1196] T 5af697387ec049778e40503779111de1 P 1d0693a9a55845948a2c382d60e351c2: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276386554690-388-0/minicluster-data/ts-1/wal/wals/5af697387ec049778e40503779111de1
I20260924 19:00:25.004431 4709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:25.004925 5019 ts_tablet_manager.cc:1939] T 5af697387ec049778e40503779111de1 P 1d0693a9a55845948a2c382d60e351c2: Deleting consensus metadata
I20260924 19:00:25.007388 4512 catalog_manager.cc:4841] TS 1d0693a9a55845948a2c382d60e351c2 (127.0.97.2:40203): tablet 5af697387ec049778e40503779111de1 (table test-table [id=b89a0983035541a2928f28136904e642]) successfully deleted
I20260924 19:00:25.092200 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 4587
I20260924 19:00:25.115612 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 4720
Sep 24 19:00:25 dist-test-slave-9fn1 krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 19:00:25.134518 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:25.116488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41642 (local address 127.0.97.2:40203)
0924 19:00:25.116691 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:25.116699 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:25.116960 (+ 261us) server_negotiation.cc:407] Connection header received
0924 19:00:25.117118 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:25.117123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:25.117214 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:25.117328 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:25.118966 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:25.120077 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:25.121881 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:25.122531 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:25.122645 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:25.127741 (+ 5096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:25.127754 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:25.127773 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:25.127819 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:25.131667 (+ 3848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:25.132299 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:25.132310 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:25.132314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:25.132469 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:25.132800 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:25.132805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:25.132809 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:25.133279 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:25.133693 (+ 414us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:25.133866 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 19:00:25.134111 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 19:00:25.135284 4842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:25.221547 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 4720
I20260924 19:00:25.238652 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 4852
Sep 24 19:00:25 dist-test-slave-9fn1 krb5kdc[4482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 19:00:25.255612 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:25.239406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43642 (local address 127.0.97.3:35609)
0924 19:00:25.239562 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:25.239567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:25.239671 (+ 104us) server_negotiation.cc:407] Connection header received
0924 19:00:25.239867 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:25.239870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:25.239936 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:25.240009 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:25.241693 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:25.242553 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:25.244326 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:25.244748 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:25.244842 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:25.250124 (+ 5282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:25.250134 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:25.250147 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:25.250181 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:25.252999 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:25.253651 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:25.253659 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:25.253667 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:25.253827 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:25.254196 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:25.254200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:25.254202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:25.254523 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:25.254824 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:25.255078 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 19:00:25.255284 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 19:00:25.256179 4974 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:25.371987 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 4852
I20260924 19:00:25.390245 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 4495
I20260924 19:00:25.391386 4556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:25.496109 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 4495
2026-09-24T19:00:25Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4107 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T19:00:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:00:25Z Disabled control of system clock
I20260924 19:00:25.560551 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46873
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 19:00:25.869943 5030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:25.870221 5030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:25.870378 5030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:25.877203 5030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:00:25.877295 5030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:25.877383 5030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:25.877460 5030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:00:25.877547 5030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:00:25.887027 5030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46873
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:25.888288 5030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:25.890089 5030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:25.901562 5038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:25.901562 5036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:25.901731 5035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:25.902434 5030 server_base.cc:1034] running on GCE node
I20260924 19:00:25.903223 5030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:25.904532 5030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:25.905853 5030 hybrid_clock.cc:648] HybridClock initialized: now 1790276425905796 us; error 51 us; skew 500 ppm
I20260924 19:00:25.906220 5030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:00:25.908295 5030 webserver.cc:460] Webserver started at http://127.0.97.62:42671/ using document root <none> and password file <none>
I20260924 19:00:25.908948 5030 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:25.909071 5030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:25.909381 5030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:25.912132 5030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "1e333da17fcb4b208b7cc2ee1839704d"
format_stamp: "Formatted at 2026-09-24 19:00:25 on dist-test-slave-9fn1"
server_key: "606122afbe92b70e8e6c9a3a1a561fc1"
server_key_iv: "4d350232f2fd8025c6d20db1ab88d9ff"
server_key_version: "encryptionkey@0"
I20260924 19:00:25.912865 5030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "1e333da17fcb4b208b7cc2ee1839704d"
format_stamp: "Formatted at 2026-09-24 19:00:25 on dist-test-slave-9fn1"
server_key: "606122afbe92b70e8e6c9a3a1a561fc1"
server_key_iv: "4d350232f2fd8025c6d20db1ab88d9ff"
server_key_version: "encryptionkey@0"
I20260924 19:00:25.917701 5030 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 19:00:25.920797 5044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:25.922092 5030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:00:25.922254 5030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "1e333da17fcb4b208b7cc2ee1839704d"
format_stamp: "Formatted at 2026-09-24 19:00:25 on dist-test-slave-9fn1"
server_key: "606122afbe92b70e8e6c9a3a1a561fc1"
server_key_iv: "4d350232f2fd8025c6d20db1ab88d9ff"
server_key_version: "encryptionkey@0"
I20260924 19:00:25.922456 5030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:25.927866 5030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:25.928593 5030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:25.928894 5030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:25.946234 5030 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46873
I20260924 19:00:25.946272 5095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46873 every 8 connection(s)
I20260924 19:00:25.947543 5030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:00:25.951040 5096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:25.957325 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 5030
I20260924 19:00:25.957559 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:00:25.958177 388 external_mini_cluster.cc:1428] Setting key 4a4b088594b89d24a446b010307c35eb
I20260924 19:00:25.961826 5096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e333da17fcb4b208b7cc2ee1839704d: Bootstrap starting.
I20260924 19:00:25.966013 5096 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e333da17fcb4b208b7cc2ee1839704d: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:25.967278 5096 log.cc:824] T 00000000000000000000000000000000 P 1e333da17fcb4b208b7cc2ee1839704d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:25.970469 5099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:25.960184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36682 (local address 127.0.97.62:46873)
0924 19:00:25.964289 (+ 4105us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:25.964310 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:25.964359 (+ 49us) server_negotiation.cc:407] Connection header received
0924 19:00:25.966675 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:25.966699 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:25.967220 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:25.967652 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:25.968627 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:25.968632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:25.968643 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:00:25.968656 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:25.968703 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:00:25.968717 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:25.968881 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:25.969059 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 19:00:25.969330 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3551,"thread_start_us":3318,"threads_started":1}
I20260924 19:00:25.970795 5096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e333da17fcb4b208b7cc2ee1839704d: No bootstrap required, opened a new log
I20260924 19:00:25.978111 5096 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e333da17fcb4b208b7cc2ee1839704d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e333da17fcb4b208b7cc2ee1839704d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46873 } }
I20260924 19:00:25.978444 5096 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e333da17fcb4b208b7cc2ee1839704d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:25.978514 5096 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e333da17fcb4b208b7cc2ee1839704d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e333da17fcb4b208b7cc2ee1839704d, State: Initialized, Role: FOLLOWER
I20260924 19:00:25.979044 5096 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e333da17fcb4b208b7cc2ee1839704d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e333da17fcb4b208b7cc2ee1839704d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46873 } }
I20260924 19:00:25.979238 5096 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e333da17fcb4b208b7cc2ee1839704d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:00:25.979355 5096 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e333da17fcb4b208b7cc2ee1839704d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:00:25.979523 5096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e333da17fcb4b208b7cc2ee1839704d [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:25.981925 5096 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e333da17fcb4b208b7cc2ee1839704d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e333da17fcb4b208b7cc2ee1839704d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46873 } }
I20260924 19:00:25.982415 5096 leader_election.cc:302] T 00000000000000000000000000000000 P 1e333da17fcb4b208b7cc2ee1839704d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e333da17fcb4b208b7cc2ee1839704d; no voters:
I20260924 19:00:25.982999 5096 leader_election.cc:288] T 00000000000000000000000000000000 P 1e333da17fcb4b208b7cc2ee1839704d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:00:25.983156 5101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e333da17fcb4b208b7cc2ee1839704d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:00:25.983889 5101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e333da17fcb4b208b7cc2ee1839704d [term 1 LEADER]: Becoming Leader. State: Replica: 1e333da17fcb4b208b7cc2ee1839704d, State: Running, Role: LEADER
I20260924 19:00:25.984452 5101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e333da17fcb4b208b7cc2ee1839704d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e333da17fcb4b208b7cc2ee1839704d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46873 } }
I20260924 19:00:25.985113 5096 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e333da17fcb4b208b7cc2ee1839704d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:00:25.987490 5103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e333da17fcb4b208b7cc2ee1839704d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e333da17fcb4b208b7cc2ee1839704d. Latest consensus state: current_term: 1 leader_uuid: "1e333da17fcb4b208b7cc2ee1839704d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e333da17fcb4b208b7cc2ee1839704d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46873 } } }
I20260924 19:00:25.987919 5103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e333da17fcb4b208b7cc2ee1839704d [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:25.988860 5109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:00:25.988783 5102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e333da17fcb4b208b7cc2ee1839704d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e333da17fcb4b208b7cc2ee1839704d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e333da17fcb4b208b7cc2ee1839704d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46873 } } }
I20260924 19:00:25.989277 5102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e333da17fcb4b208b7cc2ee1839704d [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:25.994849 5109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:00:26.003901 5109 catalog_manager.cc:1347] Generated new cluster ID: 38a97a0bcc514923a1cc8224898a3f28
I20260924 19:00:26.004019 5109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:00:26.018586 5109 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:00:26.019074 5109 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:00:26.031854 5109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e333da17fcb4b208b7cc2ee1839704d: Generated new TSK 0
I20260924 19:00:26.032863 5109 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:00:26.040712 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46873
--builtin_ntp_servers=127.0.97.20:45261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 19:00:26.365875 5120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:26.366137 5120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:26.366225 5120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:26.373346 5120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:26.373447 5120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:26.373620 5120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:00:26.382545 5120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:26.383715 5120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:26.385614 5120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:26.402590 5125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:26.402740 5126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:26.403113 5128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:26.403419 5120 server_base.cc:1034] running on GCE node
I20260924 19:00:26.404037 5120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:26.404917 5120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:26.406131 5120 hybrid_clock.cc:648] HybridClock initialized: now 1790276426406079 us; error 64 us; skew 500 ppm
I20260924 19:00:26.406440 5120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:00:26.408562 5120 webserver.cc:460] Webserver started at http://127.0.97.1:39943/ using document root <none> and password file <none>
I20260924 19:00:26.409227 5120 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:26.409351 5120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:26.409715 5120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:26.412463 5120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-0/data/instance:
uuid: "69be0cdb3cc649b9b87b073b4f671e85"
format_stamp: "Formatted at 2026-09-24 19:00:26 on dist-test-slave-9fn1"
server_key: "8491eeead62d4e4cd396ccc48174c60a"
server_key_iv: "7d274f9a10d4994572886df44f6cf33c"
server_key_version: "encryptionkey@0"
I20260924 19:00:26.413237 5120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-0/wal/instance:
uuid: "69be0cdb3cc649b9b87b073b4f671e85"
format_stamp: "Formatted at 2026-09-24 19:00:26 on dist-test-slave-9fn1"
server_key: "8491eeead62d4e4cd396ccc48174c60a"
server_key_iv: "7d274f9a10d4994572886df44f6cf33c"
server_key_version: "encryptionkey@0"
I20260924 19:00:26.418200 5120 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 19:00:26.421480 5134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:26.422809 5120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:00:26.422981 5120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "69be0cdb3cc649b9b87b073b4f671e85"
format_stamp: "Formatted at 2026-09-24 19:00:26 on dist-test-slave-9fn1"
server_key: "8491eeead62d4e4cd396ccc48174c60a"
server_key_iv: "7d274f9a10d4994572886df44f6cf33c"
server_key_version: "encryptionkey@0"
I20260924 19:00:26.423204 5120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:26.429253 5120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:26.430087 5120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:26.430349 5120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:26.431237 5120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:26.432674 5120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:26.432792 5120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:26.432914 5120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:26.432998 5120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:26.464900 5120 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38029
I20260924 19:00:26.464949 5246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38029 every 8 connection(s)
I20260924 19:00:26.466217 5120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:00:26.469022 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 5120
I20260924 19:00:26.469209 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-0/wal/instance
I20260924 19:00:26.470160 388 external_mini_cluster.cc:1428] Setting key aebbc4c0fc076466f9bce6eeab5eec20
I20260924 19:00:26.472976 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46873
--builtin_ntp_servers=127.0.97.20:45261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 19:00:26.474869 5099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:26.468729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52959 (local address 127.0.97.62:46873)
0924 19:00:26.468940 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:26.468947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:26.470608 (+ 1661us) server_negotiation.cc:407] Connection header received
0924 19:00:26.472598 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:26.472602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:26.472672 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:26.472754 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:26.474270 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:26.474275 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:26.474277 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:00:26.474292 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:26.474322 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:00:26.474332 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:26.474434 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:26.474641 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 19:00:26.474684 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 19:00:26.476109 5249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:26.469274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46873 (local address 127.0.97.1:52959)
0924 19:00:26.470424 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:26.470475 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:26.472338 (+ 1863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:26.472999 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:26.473011 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:26.473133 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:26.473861 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:00:26.473867 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:26.473918 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:00:26.474025 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:26.474430 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:26.474442 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:26.474452 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 19:00:26.474723 (+ 271us) client_negotiation.cc:240] Negotiation successful
0924 19:00:26.474950 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":355,"threads_started":1}
I20260924 19:00:26.477564 5247 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46873
I20260924 19:00:26.477828 5247 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:26.478434 5247 heartbeater.cc:511] Master 127.0.97.62:46873 requested a full tablet report, sending...
I20260924 19:00:26.480474 5061 ts_manager.cc:194] Registered new tserver with Master: 69be0cdb3cc649b9b87b073b4f671e85 (127.0.97.1:38029)
W20260924 19:00:26.791535 5250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:26.791800 5250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:26.791952 5250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:26.799280 5250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:26.799408 5250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:26.799578 5250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:00:26.808562 5250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:26.809921 5250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:26.811659 5250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:26.824767 5255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:26.824735 5258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:26.824805 5256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:26.825426 5250 server_base.cc:1034] running on GCE node
I20260924 19:00:26.826198 5250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:26.827234 5250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:26.828456 5250 hybrid_clock.cc:648] HybridClock initialized: now 1790276426828423 us; error 64 us; skew 500 ppm
I20260924 19:00:26.828778 5250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:00:26.831030 5250 webserver.cc:460] Webserver started at http://127.0.97.2:46775/ using document root <none> and password file <none>
I20260924 19:00:26.831722 5250 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:26.831856 5250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:26.832168 5250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:26.835028 5250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-1/data/instance:
uuid: "d1bfe5b69b0a4b06a6aaf159913fdad0"
format_stamp: "Formatted at 2026-09-24 19:00:26 on dist-test-slave-9fn1"
server_key: "e42c58ee2b710f9e37c1800b7c0b2d97"
server_key_iv: "8a3084abb372cb273d76f5893d08c039"
server_key_version: "encryptionkey@0"
I20260924 19:00:26.835793 5250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d1bfe5b69b0a4b06a6aaf159913fdad0"
format_stamp: "Formatted at 2026-09-24 19:00:26 on dist-test-slave-9fn1"
server_key: "e42c58ee2b710f9e37c1800b7c0b2d97"
server_key_iv: "8a3084abb372cb273d76f5893d08c039"
server_key_version: "encryptionkey@0"
I20260924 19:00:26.840664 5250 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 19:00:26.843962 5264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:26.845198 5250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:00:26.845366 5250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "d1bfe5b69b0a4b06a6aaf159913fdad0"
format_stamp: "Formatted at 2026-09-24 19:00:26 on dist-test-slave-9fn1"
server_key: "e42c58ee2b710f9e37c1800b7c0b2d97"
server_key_iv: "8a3084abb372cb273d76f5893d08c039"
server_key_version: "encryptionkey@0"
I20260924 19:00:26.845639 5250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:26.850947 5250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:26.851666 5250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:26.851925 5250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:26.852761 5250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:26.854120 5250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:26.854231 5250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:26.854354 5250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:26.854436 5250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:26.885633 5250 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34205
I20260924 19:00:26.885741 5376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34205 every 8 connection(s)
I20260924 19:00:26.887037 5250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:00:26.892021 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 5250
I20260924 19:00:26.892179 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-1/wal/instance
I20260924 19:00:26.892765 388 external_mini_cluster.cc:1428] Setting key ce0672c4015b25b41debaa21562107bd
I20260924 19:00:26.895499 5099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:26.889422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33703 (local address 127.0.97.62:46873)
0924 19:00:26.889662 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:26.889668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:26.891056 (+ 1388us) server_negotiation.cc:407] Connection header received
0924 19:00:26.893165 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:26.893169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:26.893249 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:26.893382 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:26.894872 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:26.894876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:26.894879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:00:26.894893 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:26.894923 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:00:26.894934 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:26.895033 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:26.895248 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 19:00:26.895296 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 19:00:26.895783 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46873
--builtin_ntp_servers=127.0.97.20:45261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 19:00:26.896867 5379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:26.890040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46873 (local address 127.0.97.2:33703)
0924 19:00:26.890909 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:26.890947 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:26.892892 (+ 1945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:26.893675 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:26.893688 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:26.893795 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:26.894544 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:00:26.894547 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:26.894586 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:00:26.894706 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:26.895031 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:26.895041 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:26.895049 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 19:00:26.895321 (+ 272us) client_negotiation.cc:240] Negotiation successful
0924 19:00:26.895583 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":265,"threads_started":1}
I20260924 19:00:26.898204 5377 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46873
I20260924 19:00:26.898515 5377 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:26.899190 5377 heartbeater.cc:511] Master 127.0.97.62:46873 requested a full tablet report, sending...
I20260924 19:00:26.900678 5061 ts_manager.cc:194] Registered new tserver with Master: d1bfe5b69b0a4b06a6aaf159913fdad0 (127.0.97.2:34205)
W20260924 19:00:27.215714 5380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:27.215989 5380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:27.216104 5380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:27.223188 5380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:27.223278 5380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:27.223451 5380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:00:27.232607 5380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46873
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:27.233904 5380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:27.235587 5380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:27.248241 5385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:27.248202 5388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:27.248219 5386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:27.248983 5380 server_base.cc:1034] running on GCE node
I20260924 19:00:27.249651 5380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:27.250535 5380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:27.251752 5380 hybrid_clock.cc:648] HybridClock initialized: now 1790276427251698 us; error 48 us; skew 500 ppm
I20260924 19:00:27.252050 5380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:00:27.265226 5380 webserver.cc:460] Webserver started at http://127.0.97.3:43255/ using document root <none> and password file <none>
I20260924 19:00:27.266034 5380 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:27.266168 5380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:27.266474 5380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:27.269245 5380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-2/data/instance:
uuid: "21f888e298024c6db259b949d1ac2206"
format_stamp: "Formatted at 2026-09-24 19:00:27 on dist-test-slave-9fn1"
server_key: "44ccbfa10bcc36cab88bc23c67af6e1a"
server_key_iv: "9b673ecefc67fb2dcd5582ffeebf098e"
server_key_version: "encryptionkey@0"
I20260924 19:00:27.270063 5380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-2/wal/instance:
uuid: "21f888e298024c6db259b949d1ac2206"
format_stamp: "Formatted at 2026-09-24 19:00:27 on dist-test-slave-9fn1"
server_key: "44ccbfa10bcc36cab88bc23c67af6e1a"
server_key_iv: "9b673ecefc67fb2dcd5582ffeebf098e"
server_key_version: "encryptionkey@0"
I20260924 19:00:27.274967 5380 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260924 19:00:27.278313 5394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:27.279601 5380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:00:27.279771 5380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "21f888e298024c6db259b949d1ac2206"
format_stamp: "Formatted at 2026-09-24 19:00:27 on dist-test-slave-9fn1"
server_key: "44ccbfa10bcc36cab88bc23c67af6e1a"
server_key_iv: "9b673ecefc67fb2dcd5582ffeebf098e"
server_key_version: "encryptionkey@0"
I20260924 19:00:27.279992 5380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:27.285460 5380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:27.286219 5380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:27.286470 5380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:27.287302 5380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:27.288615 5380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:27.288727 5380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:27.288846 5380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:27.288925 5380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:27.320433 5380 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33001
I20260924 19:00:27.320487 5506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33001 every 8 connection(s)
I20260924 19:00:27.321751 5380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:00:27.325011 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 5380
I20260924 19:00:27.325176 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-2/wal/instance
I20260924 19:00:27.325716 388 external_mini_cluster.cc:1428] Setting key 6ee6958b21e61ce092a1e8164d854430
I20260924 19:00:27.330001 5099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:27.324141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38663 (local address 127.0.97.62:46873)
0924 19:00:27.324351 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:27.324357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:27.325911 (+ 1554us) server_negotiation.cc:407] Connection header received
0924 19:00:27.327889 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:27.327894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:27.327960 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:27.328067 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:27.329378 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:27.329382 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:27.329385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:00:27.329400 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:27.329431 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:00:27.329443 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:27.329595 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:27.329771 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 19:00:27.329817 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 19:00:27.330948 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:27.324695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46873 (local address 127.0.97.3:38663)
0924 19:00:27.325750 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:27.325799 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:27.327575 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:27.328262 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:27.328270 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:27.328364 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:27.329108 (+ 744us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:00:27.329110 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:27.329147 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:00:27.329230 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:27.329576 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:27.329601 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:27.329607 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 19:00:27.329815 (+ 208us) client_negotiation.cc:240] Negotiation successful
0924 19:00:27.330007 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":328,"threads_started":1}
I20260924 19:00:27.332166 5507 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46873
I20260924 19:00:27.332417 5507 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:27.332914 5507 heartbeater.cc:511] Master 127.0.97.62:46873 requested a full tablet report, sending...
I20260924 19:00:27.334173 5061 ts_manager.cc:194] Registered new tserver with Master: 21f888e298024c6db259b949d1ac2206 (127.0.97.3:33001)
I20260924 19:00:27.340785 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:00:27.347553 5099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:27.345014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36696 (local address 127.0.97.62:46873)
0924 19:00:27.345197 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:27.345206 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:27.345661 (+ 455us) server_negotiation.cc:407] Connection header received
0924 19:00:27.345847 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:27.345852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:27.345905 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:27.346001 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:27.346941 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:27.346946 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:27.346951 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:00:27.346968 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:27.347005 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:00:27.347017 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:27.347135 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:27.347226 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 19:00:27.347283 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 19:00:27.351327 5061 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:00:27.361966 5061 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:00:27.392869 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:27.377718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49016 (local address 127.0.97.3:33001)
0924 19:00:27.385763 (+ 8045us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:27.385772 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:27.389330 (+ 3558us) server_negotiation.cc:407] Connection header received
0924 19:00:27.389398 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:27.389404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:27.389625 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:27.389771 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:27.391770 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:27.391779 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:27.391792 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:00:27.391810 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:27.391871 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:00:27.391889 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:27.392026 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:27.392351 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 19:00:27.392441 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7727,"thread_start_us":7219,"threads_started":1}
I20260924 19:00:27.392885 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:27.375333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34205 (local address 127.0.0.1:34280)
0924 19:00:27.385782 (+ 10449us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:27.386574 (+ 792us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:27.387394 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:27.389661 (+ 2267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:27.389669 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:27.389791 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:27.390815 (+ 1024us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:00:27.390819 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:27.390853 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:00:27.390947 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:27.391515 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:27.391525 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:27.391535 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 19:00:27.391714 (+ 179us) client_negotiation.cc:240] Negotiation successful
0924 19:00:27.392015 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10116,"thread_start_us":15330,"threads_started":1}
I20260924 19:00:27.393563 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:27.375454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34280 (local address 127.0.97.2:34205)
0924 19:00:27.375953 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:27.375959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:27.386805 (+ 10846us) server_negotiation.cc:407] Connection header received
0924 19:00:27.387610 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:27.387623 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:27.387913 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:27.388077 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:27.391153 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:27.391164 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:27.391181 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:00:27.391203 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:27.391265 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:00:27.391292 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:27.391453 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:27.393106 (+ 1653us) server_negotiation.cc:299] Negotiation successful
0924 19:00:27.393210 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":234,"threads_started":1}
I20260924 19:00:27.394122 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:27.377057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33001 (local address 127.0.0.1:49016)
0924 19:00:27.389074 (+ 12017us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:27.389091 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:27.389187 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:27.390867 (+ 1680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:27.390871 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:27.390891 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:27.391507 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:00:27.391511 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:27.391526 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:00:27.391565 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:27.392077 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:27.392082 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:27.392086 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:00:27.392227 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 19:00:27.393618 (+ 1391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11907,"thread_start_us":8480,"threads_started":1}
I20260924 19:00:27.395027 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:27.376727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38029 (local address 127.0.0.1:52546)
0924 19:00:27.387605 (+ 10878us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:27.387626 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:27.387723 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:27.389364 (+ 1641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:27.389373 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:27.389462 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:27.393618 (+ 4156us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:00:27.393621 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:27.393635 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:00:27.393680 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:27.394470 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:27.394474 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:27.394479 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:00:27.394600 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 19:00:27.394645 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10472,"thread_start_us":9270,"threads_started":1}
I20260924 19:00:27.395527 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:27.377227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52546 (local address 127.0.97.1:38029)
0924 19:00:27.387070 (+ 9843us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:27.387082 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:27.387859 (+ 777us) server_negotiation.cc:407] Connection header received
0924 19:00:27.387948 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:27.387960 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:27.388231 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:27.388397 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:27.393862 (+ 5465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:27.393871 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:27.393884 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:00:27.393904 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:27.393966 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:00:27.393989 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:27.394123 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:27.395236 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0924 19:00:27.395329 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9621,"thread_start_us":8927,"threads_started":1}
I20260924 19:00:27.397418 5312 tablet_service.cc:1467] Processing CreateTablet for tablet 966328749c414e16bb24167519c5f2af (DEFAULT_TABLE table=test-table [id=0ae8006c1f784f87be4818dbdfa8e060]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:27.397622 5442 tablet_service.cc:1467] Processing CreateTablet for tablet 966328749c414e16bb24167519c5f2af (DEFAULT_TABLE table=test-table [id=0ae8006c1f784f87be4818dbdfa8e060]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:27.398952 5312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 966328749c414e16bb24167519c5f2af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:27.398952 5442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 966328749c414e16bb24167519c5f2af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:27.405017 5182 tablet_service.cc:1467] Processing CreateTablet for tablet 966328749c414e16bb24167519c5f2af (DEFAULT_TABLE table=test-table [id=0ae8006c1f784f87be4818dbdfa8e060]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:27.406427 5182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 966328749c414e16bb24167519c5f2af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:27.414285 5525 tablet_bootstrap.cc:492] T 966328749c414e16bb24167519c5f2af P d1bfe5b69b0a4b06a6aaf159913fdad0: Bootstrap starting.
I20260924 19:00:27.420099 5526 tablet_bootstrap.cc:492] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206: Bootstrap starting.
I20260924 19:00:27.422353 5525 tablet_bootstrap.cc:654] T 966328749c414e16bb24167519c5f2af P d1bfe5b69b0a4b06a6aaf159913fdad0: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:27.423631 5526 tablet_bootstrap.cc:654] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:27.423688 5525 log.cc:824] T 966328749c414e16bb24167519c5f2af P d1bfe5b69b0a4b06a6aaf159913fdad0: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:27.424872 5526 log.cc:824] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:27.428637 5526 tablet_bootstrap.cc:492] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206: No bootstrap required, opened a new log
I20260924 19:00:27.428928 5526 ts_tablet_manager.cc:1397] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260924 19:00:27.429234 5527 tablet_bootstrap.cc:492] T 966328749c414e16bb24167519c5f2af P 69be0cdb3cc649b9b87b073b4f671e85: Bootstrap starting.
I20260924 19:00:27.432955 5527 tablet_bootstrap.cc:654] T 966328749c414e16bb24167519c5f2af P 69be0cdb3cc649b9b87b073b4f671e85: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:27.434226 5527 log.cc:824] T 966328749c414e16bb24167519c5f2af P 69be0cdb3cc649b9b87b073b4f671e85: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:27.435475 5525 tablet_bootstrap.cc:492] T 966328749c414e16bb24167519c5f2af P d1bfe5b69b0a4b06a6aaf159913fdad0: No bootstrap required, opened a new log
I20260924 19:00:27.435796 5525 ts_tablet_manager.cc:1397] T 966328749c414e16bb24167519c5f2af P d1bfe5b69b0a4b06a6aaf159913fdad0: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.005s
I20260924 19:00:27.437388 5527 tablet_bootstrap.cc:492] T 966328749c414e16bb24167519c5f2af P 69be0cdb3cc649b9b87b073b4f671e85: No bootstrap required, opened a new log
I20260924 19:00:27.437706 5527 ts_tablet_manager.cc:1397] T 966328749c414e16bb24167519c5f2af P 69be0cdb3cc649b9b87b073b4f671e85: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260924 19:00:27.440282 5526 raft_consensus.cc:348] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bfe5b69b0a4b06a6aaf159913fdad0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34205 } } peers { permanent_uuid: "69be0cdb3cc649b9b87b073b4f671e85" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38029 } } peers { permanent_uuid: "21f888e298024c6db259b949d1ac2206" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33001 } }
I20260924 19:00:27.440644 5526 raft_consensus.cc:374] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:27.440740 5526 raft_consensus.cc:729] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21f888e298024c6db259b949d1ac2206, State: Initialized, Role: FOLLOWER
I20260924 19:00:27.441323 5526 consensus_queue.cc:260] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bfe5b69b0a4b06a6aaf159913fdad0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34205 } } peers { permanent_uuid: "69be0cdb3cc649b9b87b073b4f671e85" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38029 } } peers { permanent_uuid: "21f888e298024c6db259b949d1ac2206" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33001 } }
I20260924 19:00:27.446609 5507 heartbeater.cc:503] Master 127.0.97.62:46873 was elected leader, sending a full tablet report...
I20260924 19:00:27.447120 5526 ts_tablet_manager.cc:1428] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206: Time spent starting tablet: real 0.018s user 0.014s sys 0.004s
I20260924 19:00:27.447664 5525 raft_consensus.cc:348] T 966328749c414e16bb24167519c5f2af P d1bfe5b69b0a4b06a6aaf159913fdad0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bfe5b69b0a4b06a6aaf159913fdad0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34205 } } peers { permanent_uuid: "69be0cdb3cc649b9b87b073b4f671e85" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38029 } } peers { permanent_uuid: "21f888e298024c6db259b949d1ac2206" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33001 } }
I20260924 19:00:27.448096 5525 raft_consensus.cc:374] T 966328749c414e16bb24167519c5f2af P d1bfe5b69b0a4b06a6aaf159913fdad0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:27.448220 5525 raft_consensus.cc:729] T 966328749c414e16bb24167519c5f2af P d1bfe5b69b0a4b06a6aaf159913fdad0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1bfe5b69b0a4b06a6aaf159913fdad0, State: Initialized, Role: FOLLOWER
I20260924 19:00:27.448827 5525 consensus_queue.cc:260] T 966328749c414e16bb24167519c5f2af P d1bfe5b69b0a4b06a6aaf159913fdad0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bfe5b69b0a4b06a6aaf159913fdad0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34205 } } peers { permanent_uuid: "69be0cdb3cc649b9b87b073b4f671e85" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38029 } } peers { permanent_uuid: "21f888e298024c6db259b949d1ac2206" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33001 } }
I20260924 19:00:27.448977 5527 raft_consensus.cc:348] T 966328749c414e16bb24167519c5f2af P 69be0cdb3cc649b9b87b073b4f671e85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bfe5b69b0a4b06a6aaf159913fdad0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34205 } } peers { permanent_uuid: "69be0cdb3cc649b9b87b073b4f671e85" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38029 } } peers { permanent_uuid: "21f888e298024c6db259b949d1ac2206" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33001 } }
I20260924 19:00:27.449378 5527 raft_consensus.cc:374] T 966328749c414e16bb24167519c5f2af P 69be0cdb3cc649b9b87b073b4f671e85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:27.449486 5527 raft_consensus.cc:729] T 966328749c414e16bb24167519c5f2af P 69be0cdb3cc649b9b87b073b4f671e85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69be0cdb3cc649b9b87b073b4f671e85, State: Initialized, Role: FOLLOWER
I20260924 19:00:27.449935 5377 heartbeater.cc:503] Master 127.0.97.62:46873 was elected leader, sending a full tablet report...
I20260924 19:00:27.450495 5525 ts_tablet_manager.cc:1428] T 966328749c414e16bb24167519c5f2af P d1bfe5b69b0a4b06a6aaf159913fdad0: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
I20260924 19:00:27.450356 5527 consensus_queue.cc:260] T 966328749c414e16bb24167519c5f2af P 69be0cdb3cc649b9b87b073b4f671e85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bfe5b69b0a4b06a6aaf159913fdad0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34205 } } peers { permanent_uuid: "69be0cdb3cc649b9b87b073b4f671e85" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38029 } } peers { permanent_uuid: "21f888e298024c6db259b949d1ac2206" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33001 } }
I20260924 19:00:27.451329 5247 heartbeater.cc:503] Master 127.0.97.62:46873 was elected leader, sending a full tablet report...
I20260924 19:00:27.451956 5527 ts_tablet_manager.cc:1428] T 966328749c414e16bb24167519c5f2af P 69be0cdb3cc649b9b87b073b4f671e85: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
W20260924 19:00:27.468248 5248 tablet.cc:2369] T 966328749c414e16bb24167519c5f2af P 69be0cdb3cc649b9b87b073b4f671e85: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:00:27.573354 5508 tablet.cc:2369] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:00:27.583874 5531 raft_consensus.cc:482] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:00:27.584167 5531 raft_consensus.cc:504] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bfe5b69b0a4b06a6aaf159913fdad0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34205 } } peers { permanent_uuid: "69be0cdb3cc649b9b87b073b4f671e85" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38029 } } peers { permanent_uuid: "21f888e298024c6db259b949d1ac2206" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33001 } }
I20260924 19:00:27.586107 5531 leader_election.cc:288] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1bfe5b69b0a4b06a6aaf159913fdad0 (127.0.97.2:34205), 69be0cdb3cc649b9b87b073b4f671e85 (127.0.97.1:38029)
I20260924 19:00:27.589112 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:27.586288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34205 (local address 127.0.97.3:37659)
0924 19:00:27.586604 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:27.586626 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:27.586737 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:27.587234 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:27.587239 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:27.587269 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:27.588245 (+ 976us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:00:27.588249 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:27.588267 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:00:27.588318 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:27.588681 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:27.588688 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:27.588695 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 19:00:27.588805 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 19:00:27.588857 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260924 19:00:27.589130 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:27.586526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37659 (local address 127.0.97.2:34205)
0924 19:00:27.586708 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:27.586716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:27.586740 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:00:27.586974 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:27.586984 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:27.587067 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:27.587204 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:27.588467 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:27.588473 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:27.588479 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:00:27.588499 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:27.588539 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:00:27.588558 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:27.588680 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:27.588813 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 19:00:27.588858 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 19:00:27.589632 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:27.586692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33393 (local address 127.0.97.1:38029)
0924 19:00:27.586895 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:27.586903 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:27.587706 (+ 803us) server_negotiation.cc:407] Connection header received
0924 19:00:27.587890 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:27.587894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:27.587939 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:27.588035 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:27.588947 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:27.588952 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:27.588957 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:00:27.588976 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:27.589009 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:00:27.589023 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:27.589138 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:27.589296 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 19:00:27.589365 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 19:00:27.590301 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:27.586522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38029 (local address 127.0.97.3:33393)
0924 19:00:27.587616 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:27.587628 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:27.587734 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:27.588054 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:27.588058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:27.588080 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:27.588757 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:00:27.588762 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:27.588779 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:00:27.588816 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:27.589131 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:27.589137 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:27.589142 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:00:27.589862 (+ 720us) client_negotiation.cc:240] Negotiation successful
0924 19:00:27.589914 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":969,"thread_start_us":728,"threads_started":1}
I20260924 19:00:27.590195 5332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966328749c414e16bb24167519c5f2af" candidate_uuid: "21f888e298024c6db259b949d1ac2206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1bfe5b69b0a4b06a6aaf159913fdad0" is_pre_election: true
I20260924 19:00:27.590787 5332 raft_consensus.cc:2454] T 966328749c414e16bb24167519c5f2af P d1bfe5b69b0a4b06a6aaf159913fdad0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21f888e298024c6db259b949d1ac2206 in term 0.
I20260924 19:00:27.591464 5398 leader_election.cc:302] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21f888e298024c6db259b949d1ac2206, d1bfe5b69b0a4b06a6aaf159913fdad0; no voters:
I20260924 19:00:27.591353 5202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966328749c414e16bb24167519c5f2af" candidate_uuid: "21f888e298024c6db259b949d1ac2206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69be0cdb3cc649b9b87b073b4f671e85" is_pre_election: true
I20260924 19:00:27.591856 5531 raft_consensus.cc:2781] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:00:27.591863 5202 raft_consensus.cc:2454] T 966328749c414e16bb24167519c5f2af P 69be0cdb3cc649b9b87b073b4f671e85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21f888e298024c6db259b949d1ac2206 in term 0.
I20260924 19:00:27.591998 5531 raft_consensus.cc:482] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:00:27.592104 5531 raft_consensus.cc:3037] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:27.594624 5531 raft_consensus.cc:504] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bfe5b69b0a4b06a6aaf159913fdad0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34205 } } peers { permanent_uuid: "69be0cdb3cc649b9b87b073b4f671e85" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38029 } } peers { permanent_uuid: "21f888e298024c6db259b949d1ac2206" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33001 } }
I20260924 19:00:27.595199 5531 leader_election.cc:288] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206 [CANDIDATE]: Term 1 election: Requested vote from peers d1bfe5b69b0a4b06a6aaf159913fdad0 (127.0.97.2:34205), 69be0cdb3cc649b9b87b073b4f671e85 (127.0.97.1:38029)
I20260924 19:00:27.595626 5202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966328749c414e16bb24167519c5f2af" candidate_uuid: "21f888e298024c6db259b949d1ac2206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69be0cdb3cc649b9b87b073b4f671e85"
I20260924 19:00:27.595667 5332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "966328749c414e16bb24167519c5f2af" candidate_uuid: "21f888e298024c6db259b949d1ac2206" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1bfe5b69b0a4b06a6aaf159913fdad0"
I20260924 19:00:27.595815 5202 raft_consensus.cc:3037] T 966328749c414e16bb24167519c5f2af P 69be0cdb3cc649b9b87b073b4f671e85 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:27.595850 5332 raft_consensus.cc:3037] T 966328749c414e16bb24167519c5f2af P d1bfe5b69b0a4b06a6aaf159913fdad0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:27.597971 5202 raft_consensus.cc:2454] T 966328749c414e16bb24167519c5f2af P 69be0cdb3cc649b9b87b073b4f671e85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21f888e298024c6db259b949d1ac2206 in term 1.
I20260924 19:00:27.598040 5332 raft_consensus.cc:2454] T 966328749c414e16bb24167519c5f2af P d1bfe5b69b0a4b06a6aaf159913fdad0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21f888e298024c6db259b949d1ac2206 in term 1.
I20260924 19:00:27.598390 5396 leader_election.cc:302] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21f888e298024c6db259b949d1ac2206, 69be0cdb3cc649b9b87b073b4f671e85; no voters:
I20260924 19:00:27.598687 5531 raft_consensus.cc:2781] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:00:27.599128 5531 raft_consensus.cc:686] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206 [term 1 LEADER]: Becoming Leader. State: Replica: 21f888e298024c6db259b949d1ac2206, State: Running, Role: LEADER
I20260924 19:00:27.599560 5531 consensus_queue.cc:237] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bfe5b69b0a4b06a6aaf159913fdad0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34205 } } peers { permanent_uuid: "69be0cdb3cc649b9b87b073b4f671e85" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38029 } } peers { permanent_uuid: "21f888e298024c6db259b949d1ac2206" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33001 } }
I20260924 19:00:27.603680 5061 catalog_manager.cc:5495] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21f888e298024c6db259b949d1ac2206 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "21f888e298024c6db259b949d1ac2206" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1bfe5b69b0a4b06a6aaf159913fdad0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69be0cdb3cc649b9b87b073b4f671e85" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21f888e298024c6db259b949d1ac2206" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33001 } health_report { overall_health: HEALTHY } } }
W20260924 19:00:27.639088 5378 tablet.cc:2369] T 966328749c414e16bb24167519c5f2af P d1bfe5b69b0a4b06a6aaf159913fdad0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:00:27.642520 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:27.640491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49030 (local address 127.0.97.3:33001)
0924 19:00:27.640708 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:27.640712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:27.640732 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:00:27.640918 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:27.640921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:27.640998 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:27.641085 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:27.641973 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:27.641977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:27.641980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:00:27.641994 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:27.642022 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:00:27.642033 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:27.642124 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:27.642211 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 19:00:27.642254 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 19:00:27.650264 5332 raft_consensus.cc:1261] T 966328749c414e16bb24167519c5f2af P d1bfe5b69b0a4b06a6aaf159913fdad0 [term 1 FOLLOWER]: Refusing update from remote peer 21f888e298024c6db259b949d1ac2206: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:00:27.650370 5202 raft_consensus.cc:1261] T 966328749c414e16bb24167519c5f2af P 69be0cdb3cc649b9b87b073b4f671e85 [term 1 FOLLOWER]: Refusing update from remote peer 21f888e298024c6db259b949d1ac2206: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:00:27.651171 5535 consensus_queue.cc:1035] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1bfe5b69b0a4b06a6aaf159913fdad0" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:00:27.651525 5531 consensus_queue.cc:1035] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69be0cdb3cc649b9b87b073b4f671e85" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:00:27.662768 5539 mvcc.cc:205] Tried to move back new op lower bound from 7332972247646302208 to 7332972247451734016. Current Snapshot: MvccSnapshot[applied={T|T < 7332972247646302208 or (T in {7332972247646302208})}]
I20260924 19:00:27.673766 5061 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:00:27.674026 5061 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:00:27.677279 5061 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e333da17fcb4b208b7cc2ee1839704d: Sending DeleteTablet for 3 replicas of tablet 966328749c414e16bb24167519c5f2af
I20260924 19:00:27.678409 5182 tablet_service.cc:1514] Processing DeleteTablet for tablet 966328749c414e16bb24167519c5f2af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:27 UTC) from {username='slave'} at 127.0.0.1:52546
I20260924 19:00:27.678409 5312 tablet_service.cc:1514] Processing DeleteTablet for tablet 966328749c414e16bb24167519c5f2af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:27 UTC) from {username='slave'} at 127.0.0.1:34280
I20260924 19:00:27.678490 5442 tablet_service.cc:1514] Processing DeleteTablet for tablet 966328749c414e16bb24167519c5f2af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:27 UTC) from {username='slave'} at 127.0.0.1:49016
I20260924 19:00:27.679381 5549 tablet_replica.cc:330] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206: stopping tablet replica
I20260924 19:00:27.679872 5549 raft_consensus.cc:2229] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:00:27.680632 5549 raft_consensus.cc:2258] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:27.680824 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 5120
I20260924 19:00:27.681523 5547 tablet_replica.cc:330] T 966328749c414e16bb24167519c5f2af P 69be0cdb3cc649b9b87b073b4f671e85: stopping tablet replica
I20260924 19:00:27.681968 5547 raft_consensus.cc:2229] T 966328749c414e16bb24167519c5f2af P 69be0cdb3cc649b9b87b073b4f671e85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:00:27.682585 5547 raft_consensus.cc:2258] T 966328749c414e16bb24167519c5f2af P 69be0cdb3cc649b9b87b073b4f671e85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:27.683418 5548 tablet_replica.cc:330] T 966328749c414e16bb24167519c5f2af P d1bfe5b69b0a4b06a6aaf159913fdad0: stopping tablet replica
I20260924 19:00:27.683902 5548 raft_consensus.cc:2229] T 966328749c414e16bb24167519c5f2af P d1bfe5b69b0a4b06a6aaf159913fdad0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:00:27.684489 5548 raft_consensus.cc:2258] T 966328749c414e16bb24167519c5f2af P d1bfe5b69b0a4b06a6aaf159913fdad0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:27.684788 5547 ts_tablet_manager.cc:1905] T 966328749c414e16bb24167519c5f2af P 69be0cdb3cc649b9b87b073b4f671e85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:27.686673 5548 ts_tablet_manager.cc:1905] T 966328749c414e16bb24167519c5f2af P d1bfe5b69b0a4b06a6aaf159913fdad0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:27.689550 5549 ts_tablet_manager.cc:1905] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:27.693605 5548 ts_tablet_manager.cc:1918] T 966328749c414e16bb24167519c5f2af P d1bfe5b69b0a4b06a6aaf159913fdad0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:00:27.693812 5548 log.cc:1196] T 966328749c414e16bb24167519c5f2af P d1bfe5b69b0a4b06a6aaf159913fdad0: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-1/wal/wals/966328749c414e16bb24167519c5f2af
I20260924 19:00:27.694116 5547 ts_tablet_manager.cc:1918] T 966328749c414e16bb24167519c5f2af P 69be0cdb3cc649b9b87b073b4f671e85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:00:27.694280 5547 log.cc:1196] T 966328749c414e16bb24167519c5f2af P 69be0cdb3cc649b9b87b073b4f671e85: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-0/wal/wals/966328749c414e16bb24167519c5f2af
I20260924 19:00:27.694581 5548 ts_tablet_manager.cc:1939] T 966328749c414e16bb24167519c5f2af P d1bfe5b69b0a4b06a6aaf159913fdad0: Deleting consensus metadata
I20260924 19:00:27.694757 5547 ts_tablet_manager.cc:1939] T 966328749c414e16bb24167519c5f2af P 69be0cdb3cc649b9b87b073b4f671e85: Deleting consensus metadata
I20260924 19:00:27.696485 5046 catalog_manager.cc:4841] TS 69be0cdb3cc649b9b87b073b4f671e85 (127.0.97.1:38029): tablet 966328749c414e16bb24167519c5f2af (table test-table [id=0ae8006c1f784f87be4818dbdfa8e060]) successfully deleted
I20260924 19:00:27.696537 5048 catalog_manager.cc:4841] TS d1bfe5b69b0a4b06a6aaf159913fdad0 (127.0.97.2:34205): tablet 966328749c414e16bb24167519c5f2af (table test-table [id=0ae8006c1f784f87be4818dbdfa8e060]) successfully deleted
I20260924 19:00:27.700227 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:27.682934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52556 (local address 127.0.97.1:38029)
0924 19:00:27.683120 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:27.683126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:27.698055 (+ 14929us) server_negotiation.cc:407] Connection header received
0924 19:00:27.698234 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:27.698239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:27.698334 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:27.698443 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:27.699458 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:27.699465 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:27.699469 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:00:27.699533 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:27.699569 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:00:27.699584 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:27.699722 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:27.699843 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 19:00:27.699903 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 19:00:27.700919 5242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:27.701159 5549 ts_tablet_manager.cc:1918] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:00:27.701325 5549 log.cc:1196] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276386554690-388-0/minicluster-data/ts-2/wal/wals/966328749c414e16bb24167519c5f2af
I20260924 19:00:27.702069 5549 ts_tablet_manager.cc:1939] T 966328749c414e16bb24167519c5f2af P 21f888e298024c6db259b949d1ac2206: Deleting consensus metadata
I20260924 19:00:27.703445 5047 catalog_manager.cc:4841] TS 21f888e298024c6db259b949d1ac2206 (127.0.97.3:33001): tablet 966328749c414e16bb24167519c5f2af (table test-table [id=0ae8006c1f784f87be4818dbdfa8e060]) successfully deleted
I20260924 19:00:27.788046 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 5120
I20260924 19:00:27.805212 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 5250
I20260924 19:00:27.808099 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:27.806050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34284 (local address 127.0.97.2:34205)
0924 19:00:27.806212 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:27.806216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:27.806238 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:00:27.806427 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:27.806431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:27.806515 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:27.806601 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:27.807471 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:27.807475 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:27.807478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:00:27.807491 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:27.807518 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:00:27.807530 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:27.807624 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:27.807743 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 19:00:27.807786 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 19:00:27.808732 5372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:27.895856 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 5250
I20260924 19:00:27.911971 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 5380
I20260924 19:00:27.914927 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:27.912884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49044 (local address 127.0.97.3:33001)
0924 19:00:27.913075 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:27.913083 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:27.913102 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:00:27.913190 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:27.913200 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:27.913245 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:27.913366 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:27.914263 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:27.914269 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:27.914274 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:00:27.914291 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:27.914325 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:00:27.914340 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:27.914470 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:27.914568 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 19:00:27.914629 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 19:00:27.915838 5502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:28.030980 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 5380
I20260924 19:00:28.049110 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 5030
I20260924 19:00:28.050273 5091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:28.150640 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 5030
2026-09-24T19:00:28Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2653 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 19:00:28.190928 388 mini_oidc.cc:152] Starting JWKS server
I20260924 19:00:28.193543 388 webserver.cc:460] Webserver started at https://127.0.0.1:44953/ using document root <none> and password file <none>
I20260924 19:00:28.193881 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 19:00:28.194525 388 webserver.cc:460] Webserver started at http://127.0.0.1:42871/ using document root <none> and password file <none>
2026-09-24T19:00:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:00:28Z Disabled control of system clock
I20260924 19:00:28.220409 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41127
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41127
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44953/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276386554690-388-0/testchainca.cert with env {}
W20260924 19:00:28.531214 5560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:28.531497 5560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:28.531661 5560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:28.538822 5560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:00:28.538952 5560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:28.539018 5560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:28.539112 5560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:00:28.539189 5560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 19:00:28.539299 5560 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 19:00:28.539369 5560 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44953/jwks.json
I20260924 19:00:28.549150 5560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41127
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44953/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276386554690-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:28.550477 5560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:28.551781 5560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:28.563616 5568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:28.563614 5565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:28.563685 5566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:28.564473 5560 server_base.cc:1034] running on GCE node
I20260924 19:00:28.565163 5560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:28.566592 5560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:28.567794 5560 hybrid_clock.cc:648] HybridClock initialized: now 1790276428567744 us; error 44 us; skew 500 ppm
I20260924 19:00:28.568142 5560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:00:28.570464 5560 webserver.cc:460] Webserver started at http://127.0.97.62:41823/ using document root <none> and password file <none>
I20260924 19:00:28.571152 5560 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:28.571273 5560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:28.571588 5560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:28.574378 5560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "9aca6393d1bd4d218c73ddb82b75aeb4"
format_stamp: "Formatted at 2026-09-24 19:00:28 on dist-test-slave-9fn1"
server_key: "6c40af12bd87b1f571ff9437bfaac2ad"
server_key_iv: "7783a87d30709a0ea6662537f9c93dbd"
server_key_version: "encryptionkey@0"
I20260924 19:00:28.575165 5560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "9aca6393d1bd4d218c73ddb82b75aeb4"
format_stamp: "Formatted at 2026-09-24 19:00:28 on dist-test-slave-9fn1"
server_key: "6c40af12bd87b1f571ff9437bfaac2ad"
server_key_iv: "7783a87d30709a0ea6662537f9c93dbd"
server_key_version: "encryptionkey@0"
I20260924 19:00:28.580282 5560 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 19:00:28.586217 5574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:28.588239 5560 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260924 19:00:28.588423 5560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "9aca6393d1bd4d218c73ddb82b75aeb4"
format_stamp: "Formatted at 2026-09-24 19:00:28 on dist-test-slave-9fn1"
server_key: "6c40af12bd87b1f571ff9437bfaac2ad"
server_key_iv: "7783a87d30709a0ea6662537f9c93dbd"
server_key_version: "encryptionkey@0"
I20260924 19:00:28.588661 5560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:28.611248 5560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:28.612042 5560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:28.612274 5560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:28.629326 5560 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41127
I20260924 19:00:28.629345 5625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41127 every 8 connection(s)
I20260924 19:00:28.630728 5560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:00:28.634415 5626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:28.638266 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 5560
I20260924 19:00:28.638494 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:00:28.639261 388 external_mini_cluster.cc:1428] Setting key 466a853897ad9bdf5bd5be1d9580e887
I20260924 19:00:28.646890 5626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aca6393d1bd4d218c73ddb82b75aeb4: Bootstrap starting.
I20260924 19:00:28.650085 5626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9aca6393d1bd4d218c73ddb82b75aeb4: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:28.651193 5626 log.cc:824] T 00000000000000000000000000000000 P 9aca6393d1bd4d218c73ddb82b75aeb4: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:28.652151 5629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:28.641059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45604 (local address 127.0.97.62:41127)
0924 19:00:28.642049 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:28.642060 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:28.642100 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:00:28.643629 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:28.643649 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:28.644129 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:28.644522 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:28.645103 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:28.646398 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:28.647748 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:28.648417 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:28.648613 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:28.649865 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:28.649872 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:28.649886 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:00:28.650468 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:28.650562 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:00:28.650587 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:28.650705 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:28.650924 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 19:00:28.651214 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":319,"threads_started":1}
I20260924 19:00:28.653905 5626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9aca6393d1bd4d218c73ddb82b75aeb4: No bootstrap required, opened a new log
I20260924 19:00:28.655689 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:00:28.661537 5626 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9aca6393d1bd4d218c73ddb82b75aeb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aca6393d1bd4d218c73ddb82b75aeb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41127 } }
I20260924 19:00:28.661936 5626 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9aca6393d1bd4d218c73ddb82b75aeb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:28.662043 5626 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9aca6393d1bd4d218c73ddb82b75aeb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aca6393d1bd4d218c73ddb82b75aeb4, State: Initialized, Role: FOLLOWER
I20260924 19:00:28.662608 5626 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9aca6393d1bd4d218c73ddb82b75aeb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aca6393d1bd4d218c73ddb82b75aeb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41127 } }
I20260924 19:00:28.662833 5626 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9aca6393d1bd4d218c73ddb82b75aeb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:00:28.662972 5626 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9aca6393d1bd4d218c73ddb82b75aeb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:00:28.663138 5626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9aca6393d1bd4d218c73ddb82b75aeb4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:28.665584 5626 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9aca6393d1bd4d218c73ddb82b75aeb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aca6393d1bd4d218c73ddb82b75aeb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41127 } }
I20260924 19:00:28.666047 5626 leader_election.cc:302] T 00000000000000000000000000000000 P 9aca6393d1bd4d218c73ddb82b75aeb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9aca6393d1bd4d218c73ddb82b75aeb4; no voters:
I20260924 19:00:28.666595 5626 leader_election.cc:288] T 00000000000000000000000000000000 P 9aca6393d1bd4d218c73ddb82b75aeb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:00:28.666735 5631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9aca6393d1bd4d218c73ddb82b75aeb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:00:28.667692 5631 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9aca6393d1bd4d218c73ddb82b75aeb4 [term 1 LEADER]: Becoming Leader. State: Replica: 9aca6393d1bd4d218c73ddb82b75aeb4, State: Running, Role: LEADER
I20260924 19:00:28.668251 5631 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9aca6393d1bd4d218c73ddb82b75aeb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aca6393d1bd4d218c73ddb82b75aeb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41127 } }
I20260924 19:00:28.668753 5626 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9aca6393d1bd4d218c73ddb82b75aeb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:00:28.671170 5632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aca6393d1bd4d218c73ddb82b75aeb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9aca6393d1bd4d218c73ddb82b75aeb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aca6393d1bd4d218c73ddb82b75aeb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41127 } } }
I20260924 19:00:28.671146 5633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9aca6393d1bd4d218c73ddb82b75aeb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9aca6393d1bd4d218c73ddb82b75aeb4. Latest consensus state: current_term: 1 leader_uuid: "9aca6393d1bd4d218c73ddb82b75aeb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9aca6393d1bd4d218c73ddb82b75aeb4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41127 } } }
I20260924 19:00:28.671607 5632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aca6393d1bd4d218c73ddb82b75aeb4 [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:28.671765 5633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9aca6393d1bd4d218c73ddb82b75aeb4 [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:28.674019 5639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:00:28.679138 5639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:00:28.687076 5639 catalog_manager.cc:1347] Generated new cluster ID: 7c93b87526b3499a86ff516ec52092e8
I20260924 19:00:28.687242 5639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:00:28.707250 5639 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:00:28.709088 5639 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:00:28.731962 5639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9aca6393d1bd4d218c73ddb82b75aeb4: Generated new TSK 0
I20260924 19:00:28.733029 5639 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 19:00:28.771835 5656 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 19:00:28.772001 5656 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 19:00:28.773236 5629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:28.751627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45616 (local address 127.0.97.62:41127)
0924 19:00:28.751810 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:28.751816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:28.752313 (+ 497us) server_negotiation.cc:407] Connection header received
0924 19:00:28.752595 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:28.752600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:28.752683 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:28.752831 (+ 148us) server_negotiation.cc:226] Negotiated authn=JWT
0924 19:00:28.753367 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:28.754542 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:28.755877 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:28.756568 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:28.756691 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:28.757155 (+ 464us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 19:00:28.772561 (+ 15406us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44953/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 19:00:28.772934 (+ 373us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45616: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44953/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 19:00:28.773381 5629 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45616: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44953/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 19:00:28.775013 5655 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:00:28.751513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41127 (local address 127.0.0.1:45616)
0924 19:00:28.752220 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:28.752236 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:28.752380 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:28.752875 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:28.752878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:28.752887 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0924 19:00:28.753223 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:28.753231 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:28.754661 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:28.754665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:28.755747 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:28.755757 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:28.756675 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:28.756727 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:28.756855 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:28.756908 (+ 53us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 19:00:28.772969 (+ 16061us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44953/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 19:00:28.773253 (+ 284us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:41127: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44953/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":392,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:41127: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44953/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 19:00:28.776571 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 5560
I20260924 19:00:28.777812 5621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:28.870267 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 5560
2026-09-24T19:00:28Z chronyd exiting
I20260924 19:00:28.919727 388 test_util.cc:175] -----------------------------------------------
I20260924 19:00:28.919859 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276386554690-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (733 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 19:00:28.923549 388 mini_oidc.cc:152] Starting JWKS server
I20260924 19:00:28.925336 388 webserver.cc:460] Webserver started at https://127.0.0.1:39187/ using document root <none> and password file <none>
I20260924 19:00:28.925604 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 19:00:28.926249 388 webserver.cc:460] Webserver started at http://127.0.0.1:43035/ using document root <none> and password file <none>
2026-09-24T19:00:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:00:28Z Disabled control of system clock
I20260924 19:00:28.951783 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44235
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44235
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39187/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276386554690-388-0/testchainca.cert with env {}
W20260924 19:00:29.262981 5663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:29.263268 5663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:29.263433 5663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:29.270617 5663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:00:29.270735 5663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:29.270817 5663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:29.270908 5663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:00:29.270984 5663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 19:00:29.271081 5663 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 19:00:29.271147 5663 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39187/jwks.json
I20260924 19:00:29.281102 5663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44235
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39187/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276386554690-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:29.282382 5663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:29.283658 5663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:29.295316 5671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:29.295356 5669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:29.295392 5668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:29.296172 5663 server_base.cc:1034] running on GCE node
I20260924 19:00:29.296940 5663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:29.298370 5663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:29.299564 5663 hybrid_clock.cc:648] HybridClock initialized: now 1790276429299516 us; error 36 us; skew 500 ppm
I20260924 19:00:29.299901 5663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:00:29.302143 5663 webserver.cc:460] Webserver started at http://127.0.97.62:45811/ using document root <none> and password file <none>
I20260924 19:00:29.302793 5663 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:29.302909 5663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:29.303243 5663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:29.306067 5663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "7bbf023dcda24efa90386e01241d941e"
format_stamp: "Formatted at 2026-09-24 19:00:29 on dist-test-slave-9fn1"
server_key: "88bebf5d10ca98a96467370d2d55f2f7"
server_key_iv: "c80c824132e5bad2e655720620cd5269"
server_key_version: "encryptionkey@0"
I20260924 19:00:29.306923 5663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "7bbf023dcda24efa90386e01241d941e"
format_stamp: "Formatted at 2026-09-24 19:00:29 on dist-test-slave-9fn1"
server_key: "88bebf5d10ca98a96467370d2d55f2f7"
server_key_iv: "c80c824132e5bad2e655720620cd5269"
server_key_version: "encryptionkey@0"
I20260924 19:00:29.311882 5663 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:00:29.315157 5677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:29.316457 5663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:00:29.316627 5663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "7bbf023dcda24efa90386e01241d941e"
format_stamp: "Formatted at 2026-09-24 19:00:29 on dist-test-slave-9fn1"
server_key: "88bebf5d10ca98a96467370d2d55f2f7"
server_key_iv: "c80c824132e5bad2e655720620cd5269"
server_key_version: "encryptionkey@0"
I20260924 19:00:29.316835 5663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:29.335011 5663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:29.335852 5663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:29.336143 5663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:29.354249 5663 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44235
I20260924 19:00:29.354259 5728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44235 every 8 connection(s)
I20260924 19:00:29.355580 5663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:00:29.357878 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 5663
I20260924 19:00:29.358072 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:00:29.358711 388 external_mini_cluster.cc:1428] Setting key a29495773ae0b2834e4d1d27077fd8dd
I20260924 19:00:29.359395 5729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:29.372438 5732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:29.360465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59972 (local address 127.0.97.62:44235)
0924 19:00:29.361378 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:29.361389 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:29.361428 (+ 39us) server_negotiation.cc:407] Connection header received
0924 19:00:29.363783 (+ 2355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:29.363808 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:29.364269 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:29.364600 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:29.365180 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:29.366532 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:29.368240 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:29.368874 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:29.369018 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:29.370236 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:29.370244 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:29.370261 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:00:29.370810 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:29.370889 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:00:29.370908 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:29.370995 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:29.371219 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 19:00:29.371552 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":335,"threads_started":1}
I20260924 19:00:29.372718 5729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bbf023dcda24efa90386e01241d941e: Bootstrap starting.
I20260924 19:00:29.375841 5729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bbf023dcda24efa90386e01241d941e: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:29.375998 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:00:29.377053 5729 log.cc:824] T 00000000000000000000000000000000 P 7bbf023dcda24efa90386e01241d941e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:29.379755 5729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bbf023dcda24efa90386e01241d941e: No bootstrap required, opened a new log
I20260924 19:00:29.386435 5729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bbf023dcda24efa90386e01241d941e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bbf023dcda24efa90386e01241d941e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44235 } }
I20260924 19:00:29.386735 5729 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bbf023dcda24efa90386e01241d941e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:29.386809 5729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bbf023dcda24efa90386e01241d941e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bbf023dcda24efa90386e01241d941e, State: Initialized, Role: FOLLOWER
I20260924 19:00:29.387374 5729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bbf023dcda24efa90386e01241d941e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bbf023dcda24efa90386e01241d941e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44235 } }
I20260924 19:00:29.387611 5729 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bbf023dcda24efa90386e01241d941e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:00:29.387746 5729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bbf023dcda24efa90386e01241d941e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:00:29.387904 5729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bbf023dcda24efa90386e01241d941e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:29.389926 5729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bbf023dcda24efa90386e01241d941e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bbf023dcda24efa90386e01241d941e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44235 } }
I20260924 19:00:29.390390 5729 leader_election.cc:302] T 00000000000000000000000000000000 P 7bbf023dcda24efa90386e01241d941e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bbf023dcda24efa90386e01241d941e; no voters:
I20260924 19:00:29.390902 5729 leader_election.cc:288] T 00000000000000000000000000000000 P 7bbf023dcda24efa90386e01241d941e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:00:29.391026 5734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bbf023dcda24efa90386e01241d941e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:00:29.391726 5734 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bbf023dcda24efa90386e01241d941e [term 1 LEADER]: Becoming Leader. State: Replica: 7bbf023dcda24efa90386e01241d941e, State: Running, Role: LEADER
I20260924 19:00:29.392217 5734 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bbf023dcda24efa90386e01241d941e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bbf023dcda24efa90386e01241d941e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44235 } }
I20260924 19:00:29.393018 5729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bbf023dcda24efa90386e01241d941e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:00:29.395215 5736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bbf023dcda24efa90386e01241d941e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bbf023dcda24efa90386e01241d941e. Latest consensus state: current_term: 1 leader_uuid: "7bbf023dcda24efa90386e01241d941e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bbf023dcda24efa90386e01241d941e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44235 } } }
I20260924 19:00:29.395224 5735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bbf023dcda24efa90386e01241d941e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bbf023dcda24efa90386e01241d941e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bbf023dcda24efa90386e01241d941e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44235 } } }
I20260924 19:00:29.395686 5735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bbf023dcda24efa90386e01241d941e [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:29.395687 5736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bbf023dcda24efa90386e01241d941e [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:29.400359 5740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:00:29.404471 5740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:00:29.412678 5740 catalog_manager.cc:1347] Generated new cluster ID: eb1776f12a5b4aa9b88fdca837f31658
I20260924 19:00:29.412811 5740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:00:29.427908 5740 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:00:29.429306 5740 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:00:29.440217 5740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bbf023dcda24efa90386e01241d941e: Generated new TSK 0
I20260924 19:00:29.441051 5740 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 19:00:29.474431 5759 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 19:00:29.474556 5759 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260924 19:00:29.475742 5758 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:00:29.455950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44235 (local address 127.0.0.1:59982)
0924 19:00:29.456619 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:29.456635 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:29.456755 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:29.457228 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:29.457231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:29.457240 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0924 19:00:29.457490 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:29.457541 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:29.458948 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:29.458951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:29.459974 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:29.459983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:29.460956 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:29.460992 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:29.461132 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:29.461162 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 19:00:29.475297 (+ 14135us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39187/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 19:00:29.475476 (+ 179us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:44235: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39187/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":384,"threads_started":1}
I20260924 19:00:29.475770 5732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:29.456103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59982 (local address 127.0.97.62:44235)
0924 19:00:29.456300 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:29.456308 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:29.456718 (+ 410us) server_negotiation.cc:407] Connection header received
0924 19:00:29.456932 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:29.456938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:29.457030 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:29.457205 (+ 175us) server_negotiation.cc:226] Negotiated authn=JWT
0924 19:00:29.457687 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:29.458820 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:29.460110 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:29.460827 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:29.460957 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:29.461395 (+ 438us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 19:00:29.475089 (+ 13694us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39187/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 19:00:29.475441 (+ 352us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59982: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39187/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":64}
W20260924 19:00:29.475890 5732 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59982: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39187/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260924 19:00:29.477200 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 5663
I20260924 19:00:29.478345 5724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:29.568336 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 5663
2026-09-24T19:00:29Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (699 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 19:00:29.622939 388 mini_oidc.cc:152] Starting JWKS server
I20260924 19:00:29.624754 388 webserver.cc:460] Webserver started at https://127.0.0.1:46729/ using document root <none> and password file <none>
I20260924 19:00:29.624995 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 19:00:29.625641 388 webserver.cc:460] Webserver started at http://127.0.0.1:39995/ using document root <none> and password file <none>
2026-09-24T19:00:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:00:29Z Disabled control of system clock
I20260924 19:00:29.650996 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42147
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42147
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46729/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 19:00:29.958636 5766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:29.958942 5766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:29.959092 5766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:29.966244 5766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:00:29.966367 5766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:29.966441 5766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:29.966542 5766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:00:29.966614 5766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 19:00:29.966703 5766 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 19:00:29.966761 5766 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46729/jwks.json
I20260924 19:00:29.976065 5766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42147
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46729/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:29.977551 5766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:29.979267 5766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:29.991155 5771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:29.991076 5774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:29.991114 5772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:29.991839 5766 server_base.cc:1034] running on GCE node
I20260924 19:00:29.992621 5766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:29.994083 5766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:29.995276 5766 hybrid_clock.cc:648] HybridClock initialized: now 1790276429995217 us; error 50 us; skew 500 ppm
I20260924 19:00:29.995587 5766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:00:30.003531 5766 webserver.cc:460] Webserver started at http://127.0.97.62:41747/ using document root <none> and password file <none>
I20260924 19:00:30.004237 5766 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:30.004330 5766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:30.004603 5766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:30.007579 5766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "21f1d73dc2974f25a07580072a537e71"
format_stamp: "Formatted at 2026-09-24 19:00:29 on dist-test-slave-9fn1"
server_key: "aed8bd88897622b723f324d417228c55"
server_key_iv: "b39a09a3dcb7d75909f970379a6c5166"
server_key_version: "encryptionkey@0"
I20260924 19:00:30.008320 5766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "21f1d73dc2974f25a07580072a537e71"
format_stamp: "Formatted at 2026-09-24 19:00:29 on dist-test-slave-9fn1"
server_key: "aed8bd88897622b723f324d417228c55"
server_key_iv: "b39a09a3dcb7d75909f970379a6c5166"
server_key_version: "encryptionkey@0"
I20260924 19:00:30.013316 5766 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260924 19:00:30.016785 5780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:30.018283 5766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:00:30.018455 5766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "21f1d73dc2974f25a07580072a537e71"
format_stamp: "Formatted at 2026-09-24 19:00:29 on dist-test-slave-9fn1"
server_key: "aed8bd88897622b723f324d417228c55"
server_key_iv: "b39a09a3dcb7d75909f970379a6c5166"
server_key_version: "encryptionkey@0"
I20260924 19:00:30.018678 5766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:30.047921 5766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:30.048795 5766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:30.049073 5766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:30.067049 5766 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42147
I20260924 19:00:30.067024 5831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42147 every 8 connection(s)
I20260924 19:00:30.068326 5766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:00:30.071831 5832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:30.077371 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 5766
I20260924 19:00:30.077615 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:00:30.078238 388 external_mini_cluster.cc:1428] Setting key 84f297a2a35c089d09d90efe3d08a67f
I20260924 19:00:30.082069 5832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21f1d73dc2974f25a07580072a537e71: Bootstrap starting.
I20260924 19:00:30.085424 5832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21f1d73dc2974f25a07580072a537e71: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:30.086700 5832 log.cc:824] T 00000000000000000000000000000000 P 21f1d73dc2974f25a07580072a537e71: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:30.090148 5832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21f1d73dc2974f25a07580072a537e71: No bootstrap required, opened a new log
I20260924 19:00:30.091969 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:30.079991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39214 (local address 127.0.97.62:42147)
0924 19:00:30.080892 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:30.080902 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:30.080942 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:00:30.082599 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:30.082621 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:30.083075 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:30.083392 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:30.084025 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:30.085272 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:30.087143 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:30.088270 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:30.088407 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:30.089625 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:30.089636 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:30.089652 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:00:30.090188 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:30.090297 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:00:30.090329 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:30.090416 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:30.090646 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 19:00:30.090987 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":323,"threads_started":1}
I20260924 19:00:30.095794 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:00:30.098691 5832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21f1d73dc2974f25a07580072a537e71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f1d73dc2974f25a07580072a537e71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42147 } }
I20260924 19:00:30.099063 5832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21f1d73dc2974f25a07580072a537e71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:30.099172 5832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21f1d73dc2974f25a07580072a537e71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21f1d73dc2974f25a07580072a537e71, State: Initialized, Role: FOLLOWER
I20260924 19:00:30.099687 5832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21f1d73dc2974f25a07580072a537e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f1d73dc2974f25a07580072a537e71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42147 } }
I20260924 19:00:30.099916 5832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21f1d73dc2974f25a07580072a537e71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:00:30.100050 5832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21f1d73dc2974f25a07580072a537e71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:00:30.100212 5832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21f1d73dc2974f25a07580072a537e71 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:30.102370 5832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21f1d73dc2974f25a07580072a537e71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f1d73dc2974f25a07580072a537e71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42147 } }
I20260924 19:00:30.102831 5832 leader_election.cc:302] T 00000000000000000000000000000000 P 21f1d73dc2974f25a07580072a537e71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21f1d73dc2974f25a07580072a537e71; no voters:
I20260924 19:00:30.103411 5832 leader_election.cc:288] T 00000000000000000000000000000000 P 21f1d73dc2974f25a07580072a537e71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:00:30.103559 5837 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21f1d73dc2974f25a07580072a537e71 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:00:30.104328 5837 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21f1d73dc2974f25a07580072a537e71 [term 1 LEADER]: Becoming Leader. State: Replica: 21f1d73dc2974f25a07580072a537e71, State: Running, Role: LEADER
I20260924 19:00:30.104811 5837 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21f1d73dc2974f25a07580072a537e71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f1d73dc2974f25a07580072a537e71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42147 } }
I20260924 19:00:30.105376 5832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21f1d73dc2974f25a07580072a537e71 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:00:30.107663 5838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21f1d73dc2974f25a07580072a537e71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21f1d73dc2974f25a07580072a537e71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f1d73dc2974f25a07580072a537e71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42147 } } }
I20260924 19:00:30.107556 5839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21f1d73dc2974f25a07580072a537e71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21f1d73dc2974f25a07580072a537e71. Latest consensus state: current_term: 1 leader_uuid: "21f1d73dc2974f25a07580072a537e71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21f1d73dc2974f25a07580072a537e71" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42147 } } }
I20260924 19:00:30.108032 5838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21f1d73dc2974f25a07580072a537e71 [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:30.108038 5839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21f1d73dc2974f25a07580072a537e71 [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:30.117018 5845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:00:30.122691 5845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:00:30.131258 5845 catalog_manager.cc:1347] Generated new cluster ID: b3b6d92f986146e78b449876d90b8bb9
I20260924 19:00:30.131385 5845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:00:30.141625 5845 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:00:30.142832 5845 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:00:30.150782 5845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21f1d73dc2974f25a07580072a537e71: Generated new TSK 0
I20260924 19:00:30.151768 5845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:00:30.179487 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:30.171543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39224 (local address 127.0.97.62:42147)
0924 19:00:30.171759 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:30.171781 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:30.172230 (+ 449us) server_negotiation.cc:407] Connection header received
0924 19:00:30.172462 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:30.172465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:30.172534 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:30.172654 (+ 120us) server_negotiation.cc:226] Negotiated authn=JWT
0924 19:00:30.173203 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:30.173972 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:30.175966 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:30.176407 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:30.176500 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:30.177041 (+ 541us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 19:00:30.178843 (+ 1802us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46729/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 19:00:30.179163 (+ 320us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39224: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46729/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 19:00:30.179677 5861 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:00:30.171487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42147 (local address 127.0.0.1:39224)
0924 19:00:30.172123 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:30.172143 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:30.172297 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:30.172695 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:30.172700 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:30.172713 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0924 19:00:30.173053 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:30.173066 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:30.174110 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:30.174116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:30.175830 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:30.175846 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:30.176519 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:30.176598 (+ 79us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:30.176781 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:30.176828 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 19:00:30.179029 (+ 2201us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46729/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 19:00:30.179247 (+ 218us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:42147: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46729/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":318,"threads_started":1}
W20260924 19:00:30.179984 5835 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39224: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46729/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260924 19:00:30.181197 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 5766
I20260924 19:00:30.182193 5827 generic_service.cc:196] Checking for leaks (request via RPC)
E20260924 19:00:30.191658 5862 webserver.cc:533] Webserver: sslize failed:
E20260924 19:00:30.191773 5862 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 19:00:30.270850 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 5766
2026-09-24T19:00:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (702 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T19:00:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:00:30Z Disabled control of system clock
I20260924 19:00:30.350263 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32909
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32909
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276386554690-388-0/testchainca.cert with env {}
W20260924 19:00:30.671639 5867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:30.671911 5867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:30.672019 5867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:30.679095 5867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:00:30.679208 5867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:30.679246 5867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:30.679302 5867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:00:30.679339 5867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 19:00:30.679401 5867 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 19:00:30.679441 5867 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 19:00:30.688982 5867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32909
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276386554690-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:30.690179 5867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:30.691402 5867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:30.703255 5875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:30.703254 5873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:30.703735 5872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:30.704016 5867 server_base.cc:1034] running on GCE node
I20260924 19:00:30.704700 5867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:30.705997 5867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:30.707265 5867 hybrid_clock.cc:648] HybridClock initialized: now 1790276430707248 us; error 53 us; skew 500 ppm
I20260924 19:00:30.707590 5867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:00:30.709707 5867 webserver.cc:460] Webserver started at http://127.0.97.62:38941/ using document root <none> and password file <none>
I20260924 19:00:30.710350 5867 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:30.710479 5867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:30.710780 5867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:30.713369 5867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "009251e834854c7fba9b3689495d0a5c"
format_stamp: "Formatted at 2026-09-24 19:00:30 on dist-test-slave-9fn1"
server_key: "786801bc67cb22e6c901cbeb959aa00a"
server_key_iv: "d1a6b9a504ea64fb8c77a14e70aed3a4"
server_key_version: "encryptionkey@0"
I20260924 19:00:30.714186 5867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "009251e834854c7fba9b3689495d0a5c"
format_stamp: "Formatted at 2026-09-24 19:00:30 on dist-test-slave-9fn1"
server_key: "786801bc67cb22e6c901cbeb959aa00a"
server_key_iv: "d1a6b9a504ea64fb8c77a14e70aed3a4"
server_key_version: "encryptionkey@0"
I20260924 19:00:30.718916 5867 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 19:00:30.721980 5881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:30.723057 5867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260924 19:00:30.723224 5867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "009251e834854c7fba9b3689495d0a5c"
format_stamp: "Formatted at 2026-09-24 19:00:30 on dist-test-slave-9fn1"
server_key: "786801bc67cb22e6c901cbeb959aa00a"
server_key_iv: "d1a6b9a504ea64fb8c77a14e70aed3a4"
server_key_version: "encryptionkey@0"
I20260924 19:00:30.723415 5867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:30.738967 5867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:30.739694 5867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:30.739964 5867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:30.757759 5867 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32909
I20260924 19:00:30.757766 5932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32909 every 8 connection(s)
I20260924 19:00:30.759089 5867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:00:30.763160 5933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:30.767491 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 5867
I20260924 19:00:30.767678 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:00:30.768323 388 external_mini_cluster.cc:1428] Setting key 52422b964de108cce32be1c1bfb08a20
I20260924 19:00:30.773618 5933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 009251e834854c7fba9b3689495d0a5c: Bootstrap starting.
I20260924 19:00:30.776476 5933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 009251e834854c7fba9b3689495d0a5c: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:30.777546 5933 log.cc:824] T 00000000000000000000000000000000 P 009251e834854c7fba9b3689495d0a5c: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:30.780395 5933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 009251e834854c7fba9b3689495d0a5c: No bootstrap required, opened a new log
I20260924 19:00:30.780879 5936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:30.770110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39918 (local address 127.0.97.62:32909)
0924 19:00:30.771066 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:30.771077 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:30.771118 (+ 41us) server_negotiation.cc:407] Connection header received
0924 19:00:30.772729 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:30.772749 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:30.773187 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:30.773586 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:30.774113 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:30.775361 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:30.776595 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:30.777257 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:30.777446 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:30.778671 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:30.778683 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:30.778704 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:00:30.779230 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:30.779304 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:00:30.779323 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:30.779407 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:30.779674 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 19:00:30.779977 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":315,"threads_started":1}
I20260924 19:00:30.784524 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:00:30.788326 5933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 009251e834854c7fba9b3689495d0a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009251e834854c7fba9b3689495d0a5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32909 } }
I20260924 19:00:30.788651 5933 raft_consensus.cc:374] T 00000000000000000000000000000000 P 009251e834854c7fba9b3689495d0a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:30.788748 5933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 009251e834854c7fba9b3689495d0a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009251e834854c7fba9b3689495d0a5c, State: Initialized, Role: FOLLOWER
I20260924 19:00:30.789332 5933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 009251e834854c7fba9b3689495d0a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009251e834854c7fba9b3689495d0a5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32909 } }
I20260924 19:00:30.789602 5933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 009251e834854c7fba9b3689495d0a5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:00:30.789736 5933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 009251e834854c7fba9b3689495d0a5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:00:30.789914 5933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 009251e834854c7fba9b3689495d0a5c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:30.791966 5933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 009251e834854c7fba9b3689495d0a5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009251e834854c7fba9b3689495d0a5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32909 } }
I20260924 19:00:30.792496 5933 leader_election.cc:302] T 00000000000000000000000000000000 P 009251e834854c7fba9b3689495d0a5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 009251e834854c7fba9b3689495d0a5c; no voters:
I20260924 19:00:30.793013 5933 leader_election.cc:288] T 00000000000000000000000000000000 P 009251e834854c7fba9b3689495d0a5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:00:30.793141 5938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 009251e834854c7fba9b3689495d0a5c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:00:30.794361 5938 raft_consensus.cc:686] T 00000000000000000000000000000000 P 009251e834854c7fba9b3689495d0a5c [term 1 LEADER]: Becoming Leader. State: Replica: 009251e834854c7fba9b3689495d0a5c, State: Running, Role: LEADER
I20260924 19:00:30.794979 5938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 009251e834854c7fba9b3689495d0a5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009251e834854c7fba9b3689495d0a5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32909 } }
I20260924 19:00:30.795076 5933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 009251e834854c7fba9b3689495d0a5c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:00:30.797670 5940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 009251e834854c7fba9b3689495d0a5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 009251e834854c7fba9b3689495d0a5c. Latest consensus state: current_term: 1 leader_uuid: "009251e834854c7fba9b3689495d0a5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009251e834854c7fba9b3689495d0a5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32909 } } }
I20260924 19:00:30.797643 5939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 009251e834854c7fba9b3689495d0a5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "009251e834854c7fba9b3689495d0a5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009251e834854c7fba9b3689495d0a5c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32909 } } }
I20260924 19:00:30.798112 5939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 009251e834854c7fba9b3689495d0a5c [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:30.798112 5940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 009251e834854c7fba9b3689495d0a5c [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:30.800333 5944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:00:30.804987 5944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:00:30.815555 5944 catalog_manager.cc:1347] Generated new cluster ID: 7fb5f4c63bf840dea193b79a26ec0722
I20260924 19:00:30.815675 5944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:00:30.848090 5944 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:00:30.849905 5944 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:00:30.864279 5944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 009251e834854c7fba9b3689495d0a5c: Generated new TSK 0
I20260924 19:00:30.865305 5944 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:00:30.935093 5936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:30.880786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39926 (local address 127.0.97.62:32909)
0924 19:00:30.880964 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:30.880970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:30.881332 (+ 362us) server_negotiation.cc:407] Connection header received
0924 19:00:30.881543 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:30.881549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:30.881648 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:30.881810 (+ 162us) server_negotiation.cc:226] Negotiated authn=JWT
0924 19:00:30.882214 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:30.883364 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:30.884666 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:30.885312 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:30.885433 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:30.885940 (+ 507us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 19:00:30.934325 (+ 48385us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 19:00:30.934769 (+ 444us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39926: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 19:00:30.935140 5962 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:00:30.880642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32909 (local address 127.0.0.1:39926)
0924 19:00:30.881232 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:30.881251 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:30.881360 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:30.881824 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:30.881827 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:30.881836 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0924 19:00:30.882074 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:30.882081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:30.883494 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:30.883498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:30.884528 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:30.884537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:30.885426 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:30.885467 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:30.885647 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:30.885677 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 19:00:30.934601 (+ 48924us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 19:00:30.934854 (+ 253us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:32909: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":309,"threads_started":1}
W20260924 19:00:30.935225 5936 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39926: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260924 19:00:30.936721 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 5867
I20260924 19:00:30.937877 5928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:31.035861 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 5867
2026-09-24T19:00:31Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (744 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (393 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (399 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:00:31 dist-test-slave-9fn1 krb5kdc[5977](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:00:31 dist-test-slave-9fn1 krb5kdc[5977](info): set up 2 sockets
Sep 24 19:00:31 dist-test-slave-9fn1 krb5kdc[5977](info): commencing operation
krb5kdc: starting...
W20260924 19:00:32.906245 388 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.007s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:00:33 dist-test-slave-9fn1 krb5kdc[5977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:00:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:00:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 19:00:33.102885 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45059
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45059
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:33.415256 5990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:33.415597 5990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:33.415759 5990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:33.422730 5990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:00:33.422874 5990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:33.422981 5990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:33.423084 5990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:00:33.423171 5990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:00:33.432624 5990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:33.434002 5990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:33.435268 5990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:33.446548 5996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:33.446760 5995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:33.446760 5998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:33.447701 5990 server_base.cc:1034] running on GCE node
I20260924 19:00:33.448498 5990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:33.449851 5990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:33.451045 5990 hybrid_clock.cc:648] HybridClock initialized: now 1790276433450998 us; error 36 us; skew 500 ppm
I20260924 19:00:33.451375 5990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:33 dist-test-slave-9fn1 krb5kdc[5977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:33.454247 5990 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:00:33.455605 5990 webserver.cc:460] Webserver started at http://127.0.97.62:34007/ using document root <none> and password file <none>
I20260924 19:00:33.456216 5990 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:33.456311 5990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:33.456561 5990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:33.459381 5990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "9afe5f322c864e0384627df813be6457"
format_stamp: "Formatted at 2026-09-24 19:00:33 on dist-test-slave-9fn1"
server_key: "b8288a3134e07691024b331dbea5f27b"
server_key_iv: "883a1026da5f0d012c1f85ffccb3d9da"
server_key_version: "encryptionkey@0"
I20260924 19:00:33.460096 5990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "9afe5f322c864e0384627df813be6457"
format_stamp: "Formatted at 2026-09-24 19:00:33 on dist-test-slave-9fn1"
server_key: "b8288a3134e07691024b331dbea5f27b"
server_key_iv: "883a1026da5f0d012c1f85ffccb3d9da"
server_key_version: "encryptionkey@0"
I20260924 19:00:33.464864 5990 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260924 19:00:33.468152 6005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:33.469422 5990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:00:33.469638 5990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "9afe5f322c864e0384627df813be6457"
format_stamp: "Formatted at 2026-09-24 19:00:33 on dist-test-slave-9fn1"
server_key: "b8288a3134e07691024b331dbea5f27b"
server_key_iv: "883a1026da5f0d012c1f85ffccb3d9da"
server_key_version: "encryptionkey@0"
I20260924 19:00:33.469830 5990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:33.486063 5990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:33.489810 5990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:33.490077 5990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:33.508332 5990 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45059
I20260924 19:00:33.508366 6056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45059 every 8 connection(s)
I20260924 19:00:33.509773 5990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:00:33.511329 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 5990
I20260924 19:00:33.511529 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:00:33.512319 388 external_mini_cluster.cc:1428] Setting key 9202a01b1eca5cbb28611937948fd851
I20260924 19:00:33.514339 6057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 19:00:33 dist-test-slave-9fn1 krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:33.527778 6057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9afe5f322c864e0384627df813be6457: Bootstrap starting.
I20260924 19:00:33.531973 6057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9afe5f322c864e0384627df813be6457: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:33.533298 6057 log.cc:824] T 00000000000000000000000000000000 P 9afe5f322c864e0384627df813be6457: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:33.536034 6060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:33.514486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38550 (local address 127.0.97.62:45059)
0924 19:00:33.515265 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:33.515276 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:33.515317 (+ 41us) server_negotiation.cc:407] Connection header received
0924 19:00:33.516895 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:33.516916 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:33.517292 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:33.517691 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:33.519378 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:33.520675 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:33.522405 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:33.523043 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:33.523163 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:33.527690 (+ 4527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:33.527700 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:33.527719 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:33.527774 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:33.531561 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:33.532433 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:33.532474 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:33.532488 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:33.532722 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:33.533233 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:33.533238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:33.533241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:33.533888 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:33.534236 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:33.534647 (+ 411us) server_negotiation.cc:299] Negotiation successful
0924 19:00:33.535074 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":345,"threads_started":1}
I20260924 19:00:33.536173 6057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9afe5f322c864e0384627df813be6457: No bootstrap required, opened a new log
I20260924 19:00:33.543408 6057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9afe5f322c864e0384627df813be6457 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9afe5f322c864e0384627df813be6457" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45059 } }
I20260924 19:00:33.543751 6057 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9afe5f322c864e0384627df813be6457 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:33.543865 6057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9afe5f322c864e0384627df813be6457 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9afe5f322c864e0384627df813be6457, State: Initialized, Role: FOLLOWER
I20260924 19:00:33.544498 6057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9afe5f322c864e0384627df813be6457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9afe5f322c864e0384627df813be6457" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45059 } }
I20260924 19:00:33.544835 6057 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9afe5f322c864e0384627df813be6457 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:00:33.545009 6057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9afe5f322c864e0384627df813be6457 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:00:33.545199 6057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9afe5f322c864e0384627df813be6457 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:33.547266 6057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9afe5f322c864e0384627df813be6457 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9afe5f322c864e0384627df813be6457" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45059 } }
I20260924 19:00:33.547736 6057 leader_election.cc:302] T 00000000000000000000000000000000 P 9afe5f322c864e0384627df813be6457 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9afe5f322c864e0384627df813be6457; no voters:
I20260924 19:00:33.548261 6057 leader_election.cc:288] T 00000000000000000000000000000000 P 9afe5f322c864e0384627df813be6457 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:00:33.548408 6062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9afe5f322c864e0384627df813be6457 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:00:33.549805 6062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9afe5f322c864e0384627df813be6457 [term 1 LEADER]: Becoming Leader. State: Replica: 9afe5f322c864e0384627df813be6457, State: Running, Role: LEADER
I20260924 19:00:33.550266 6057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9afe5f322c864e0384627df813be6457 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:00:33.550266 6062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9afe5f322c864e0384627df813be6457 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9afe5f322c864e0384627df813be6457" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45059 } }
I20260924 19:00:33.553005 6063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9afe5f322c864e0384627df813be6457 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9afe5f322c864e0384627df813be6457" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9afe5f322c864e0384627df813be6457" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45059 } } }
I20260924 19:00:33.553030 6064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9afe5f322c864e0384627df813be6457 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9afe5f322c864e0384627df813be6457. Latest consensus state: current_term: 1 leader_uuid: "9afe5f322c864e0384627df813be6457" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9afe5f322c864e0384627df813be6457" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45059 } } }
I20260924 19:00:33.553435 6063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9afe5f322c864e0384627df813be6457 [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:33.553435 6064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9afe5f322c864e0384627df813be6457 [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:33.555013 6070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:00:33.560127 6070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:00:33.568361 6070 catalog_manager.cc:1347] Generated new cluster ID: 78899c8ddaee41b0bec9df8eddc8b0a3
I20260924 19:00:33.568478 6070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:00:33.576308 6070 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:00:33.577481 6070 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:00:33.587532 6070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9afe5f322c864e0384627df813be6457: Generated new TSK 0
I20260924 19:00:33.588249 6070 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 19:00:33.643323 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45059
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:33.985117 6083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:33.985384 6083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:33.985558 6083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:33.993790 6083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:33.993952 6083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:33.994138 6083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:00:34.003865 6083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:34.005071 6083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:34.006382 6083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:34.018985 6088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:34.019068 6089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:34.019220 6091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:34.019974 6083 server_base.cc:1034] running on GCE node
I20260924 19:00:34.020633 6083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:34.021616 6083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:34.022832 6083 hybrid_clock.cc:648] HybridClock initialized: now 1790276434022781 us; error 59 us; skew 500 ppm
I20260924 19:00:34.023144 6083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:34 dist-test-slave-9fn1 krb5kdc[5977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:34.026273 6083 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 19:00:34.027591 6083 webserver.cc:460] Webserver started at http://127.0.97.1:36431/ using document root <none> and password file <none>
I20260924 19:00:34.028236 6083 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:34.028378 6083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:34.028672 6083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:34.031389 6083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-0/data/instance:
uuid: "dddef438d69c479ea980ecc7ba07be8a"
format_stamp: "Formatted at 2026-09-24 19:00:34 on dist-test-slave-9fn1"
server_key: "475d23164bb5620334a6a76ee85c6221"
server_key_iv: "6120aad21549385865ebdcb3ab59d930"
server_key_version: "encryptionkey@0"
I20260924 19:00:34.032087 6083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-0/wal/instance:
uuid: "dddef438d69c479ea980ecc7ba07be8a"
format_stamp: "Formatted at 2026-09-24 19:00:34 on dist-test-slave-9fn1"
server_key: "475d23164bb5620334a6a76ee85c6221"
server_key_iv: "6120aad21549385865ebdcb3ab59d930"
server_key_version: "encryptionkey@0"
I20260924 19:00:34.037273 6083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 19:00:34.040588 6098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:34.041939 6083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:00:34.042101 6083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "dddef438d69c479ea980ecc7ba07be8a"
format_stamp: "Formatted at 2026-09-24 19:00:34 on dist-test-slave-9fn1"
server_key: "475d23164bb5620334a6a76ee85c6221"
server_key_iv: "6120aad21549385865ebdcb3ab59d930"
server_key_version: "encryptionkey@0"
I20260924 19:00:34.042343 6083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:34.057041 6083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:34.060624 6083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:34.060853 6083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:34.061774 6083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:34.062951 6083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:34.063021 6083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:34.063102 6083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:34.063145 6083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:34.094906 6083 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34435
I20260924 19:00:34.094947 6210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34435 every 8 connection(s)
I20260924 19:00:34.096194 6083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:00:34.105792 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 6083
I20260924 19:00:34.105988 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-0/wal/instance
I20260924 19:00:34.106597 388 external_mini_cluster.cc:1428] Setting key 6d77093c619f48291e8c8d44c276480b
Sep 24 19:00:34 dist-test-slave-9fn1 krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:34.115496 6213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:34.098658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49087 (local address 127.0.97.62:45059)
0924 19:00:34.099095 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:34.099100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:34.099933 (+ 833us) server_negotiation.cc:407] Connection header received
0924 19:00:34.101706 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:34.101712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:34.101818 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:34.101943 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:34.103376 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:34.104204 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:34.105722 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:34.106231 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:34.106368 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:34.109703 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:34.109709 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:34.109714 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:34.109753 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:34.112240 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:34.112940 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:34.112952 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:34.112956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:34.113108 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:34.113560 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:34.113567 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:34.113572 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:34.113933 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:34.114105 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:34.114761 (+ 656us) server_negotiation.cc:299] Negotiation successful
0924 19:00:34.114995 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":237,"threads_started":1}
I20260924 19:00:34.116063 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:34.099096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45059 (local address 127.0.97.1:49087)
0924 19:00:34.099780 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:34.099817 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:34.101394 (+ 1577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:34.102207 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:34.102215 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:34.102697 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:34.103217 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:34.103233 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:34.104361 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:34.104365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:34.105590 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:34.105602 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:34.106451 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:34.106460 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:34.106714 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:34.107403 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:34.107425 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:34.109463 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:34.112380 (+ 2917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:34.112389 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:34.112404 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:34.112814 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:34.113232 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:34.113236 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:34.113238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:34.113411 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:34.114069 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:34.114074 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:34.114364 (+ 290us) client_negotiation.cc:769] Sending connection context
0924 19:00:34.114608 (+ 244us) client_negotiation.cc:240] Negotiation successful
0924 19:00:34.114960 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":283,"threads_started":1}
I20260924 19:00:34.117606 6211 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45059
I20260924 19:00:34.117940 6211 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:34.118789 6211 heartbeater.cc:511] Master 127.0.97.62:45059 requested a full tablet report, sending...
I20260924 19:00:34.120836 6022 ts_manager.cc:194] Registered new tserver with Master: dddef438d69c479ea980ecc7ba07be8a (127.0.97.1:34435)
I20260924 19:00:34.122819 6022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49087
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 19:00:34.153544 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45059
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:34.471114 6217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:34.471395 6217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:34.471486 6217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:34.478710 6217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:34.478822 6217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:34.478931 6217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:00:34.488371 6217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:34.489573 6217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:34.490787 6217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:34.504508 6223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:34.504410 6222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:34.504463 6225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:34.505319 6217 server_base.cc:1034] running on GCE node
I20260924 19:00:34.505975 6217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:34.506923 6217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:34.508140 6217 hybrid_clock.cc:648] HybridClock initialized: now 1790276434508092 us; error 35 us; skew 500 ppm
I20260924 19:00:34.508543 6217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:34 dist-test-slave-9fn1 krb5kdc[5977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:34.512202 6217 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 19:00:34.523469 6217 webserver.cc:460] Webserver started at http://127.0.97.2:36967/ using document root <none> and password file <none>
I20260924 19:00:34.524176 6217 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:34.524322 6217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:34.524626 6217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:34.527913 6217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-1/data/instance:
uuid: "5854cb13549149009bc1a71ec7dce102"
format_stamp: "Formatted at 2026-09-24 19:00:34 on dist-test-slave-9fn1"
server_key: "494e73b8aa2d2c20f0e36c2681a9d75e"
server_key_iv: "a95e22a6d5872543a1989e85df6eaa04"
server_key_version: "encryptionkey@0"
I20260924 19:00:34.528683 6217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5854cb13549149009bc1a71ec7dce102"
format_stamp: "Formatted at 2026-09-24 19:00:34 on dist-test-slave-9fn1"
server_key: "494e73b8aa2d2c20f0e36c2681a9d75e"
server_key_iv: "a95e22a6d5872543a1989e85df6eaa04"
server_key_version: "encryptionkey@0"
I20260924 19:00:34.533834 6217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260924 19:00:34.537032 6232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:34.538367 6217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:00:34.538537 6217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "5854cb13549149009bc1a71ec7dce102"
format_stamp: "Formatted at 2026-09-24 19:00:34 on dist-test-slave-9fn1"
server_key: "494e73b8aa2d2c20f0e36c2681a9d75e"
server_key_iv: "a95e22a6d5872543a1989e85df6eaa04"
server_key_version: "encryptionkey@0"
I20260924 19:00:34.538733 6217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:34.555320 6217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:34.559495 6217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:34.559775 6217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:34.560675 6217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:34.562119 6217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:34.562218 6217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:34.562372 6217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:34.562443 6217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:34.597743 6217 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42311
I20260924 19:00:34.597765 6344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42311 every 8 connection(s)
I20260924 19:00:34.599040 6217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:00:34.602237 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 6217
I20260924 19:00:34.602427 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-1/wal/instance
I20260924 19:00:34.603008 388 external_mini_cluster.cc:1428] Setting key 636459928007060adac9460cab83fd74
Sep 24 19:00:34 dist-test-slave-9fn1 krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:34.620389 6213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:34.601477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36325 (local address 127.0.97.62:45059)
0924 19:00:34.601678 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:34.601683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:34.603096 (+ 1413us) server_negotiation.cc:407] Connection header received
0924 19:00:34.605149 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:34.605154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:34.605230 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:34.605346 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:34.606864 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:34.607552 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:34.609079 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:34.609548 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:34.609685 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:34.613522 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:34.613532 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:34.613540 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:34.613583 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:34.617014 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:34.617804 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:34.617812 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:34.617819 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:34.617972 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:34.618384 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:34.618392 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:34.618398 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:34.618781 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:34.618932 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:34.619685 (+ 753us) server_negotiation.cc:299] Negotiation successful
0924 19:00:34.619927 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 19:00:34.621075 6347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:34.602081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45059 (local address 127.0.97.2:36325)
0924 19:00:34.602928 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:34.602970 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:34.604899 (+ 1929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:34.605628 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:34.605636 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:34.606132 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:34.606705 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:34.606720 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:34.607697 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:34.607703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:34.608936 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:34.608947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:34.609769 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:34.609777 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:34.610024 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:34.610710 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:34.610732 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:34.613273 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:34.617155 (+ 3882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:34.617162 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:34.617175 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:34.617677 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:34.618081 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:34.618085 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:34.618087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:34.618259 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:34.618897 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:34.618903 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:34.619224 (+ 321us) client_negotiation.cc:769] Sending connection context
0924 19:00:34.619540 (+ 316us) client_negotiation.cc:240] Negotiation successful
0924 19:00:34.619935 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":390,"threads_started":1}
I20260924 19:00:34.622579 6345 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45059
I20260924 19:00:34.622889 6345 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 19:00:34.623679 6345 heartbeater.cc:511] Master 127.0.97.62:45059 requested a full tablet report, sending...
I20260924 19:00:34.625135 6022 ts_manager.cc:194] Registered new tserver with Master: 5854cb13549149009bc1a71ec7dce102 (127.0.97.2:42311)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260924 19:00:34.626070 6022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 19:00:34.648550 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45059
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:34.961899 6350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:34.962157 6350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:34.962340 6350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:34.969452 6350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:34.969660 6350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:34.969856 6350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:00:34.979605 6350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:34.980863 6350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:34.982211 6350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:34.997318 6355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:34.997283 6356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:34.997355 6358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:34.998442 6350 server_base.cc:1034] running on GCE node
I20260924 19:00:34.999090 6350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:35.000025 6350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:35.001245 6350 hybrid_clock.cc:648] HybridClock initialized: now 1790276435001184 us; error 57 us; skew 500 ppm
I20260924 19:00:35.001624 6350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:35 dist-test-slave-9fn1 krb5kdc[5977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:35.009811 6350 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 19:00:35.011335 6350 webserver.cc:460] Webserver started at http://127.0.97.3:38617/ using document root <none> and password file <none>
I20260924 19:00:35.012051 6350 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:35.012190 6350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:35.012496 6350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:35.015493 6350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-2/data/instance:
uuid: "cb0a670fdb814004844ec46ec2a070cc"
format_stamp: "Formatted at 2026-09-24 19:00:35 on dist-test-slave-9fn1"
server_key: "0a5d4c2bf839949b165f892d45b19337"
server_key_iv: "9e153fac04952908b6f0b02557ee983c"
server_key_version: "encryptionkey@0"
I20260924 19:00:35.016278 6350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-2/wal/instance:
uuid: "cb0a670fdb814004844ec46ec2a070cc"
format_stamp: "Formatted at 2026-09-24 19:00:35 on dist-test-slave-9fn1"
server_key: "0a5d4c2bf839949b165f892d45b19337"
server_key_iv: "9e153fac04952908b6f0b02557ee983c"
server_key_version: "encryptionkey@0"
I20260924 19:00:35.021461 6350 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 19:00:35.024672 6365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:35.026049 6350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:00:35.026209 6350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "cb0a670fdb814004844ec46ec2a070cc"
format_stamp: "Formatted at 2026-09-24 19:00:35 on dist-test-slave-9fn1"
server_key: "0a5d4c2bf839949b165f892d45b19337"
server_key_iv: "9e153fac04952908b6f0b02557ee983c"
server_key_version: "encryptionkey@0"
I20260924 19:00:35.026405 6350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:35.070254 6350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:35.074508 6350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:35.074817 6350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:35.075750 6350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:35.077065 6350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:35.077167 6350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:35.077306 6350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:35.077380 6350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:35.111310 6350 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44923
I20260924 19:00:35.111351 6477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44923 every 8 connection(s)
I20260924 19:00:35.112514 6350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:00:35.118816 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 6350
I20260924 19:00:35.119000 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/minicluster-data/ts-2/wal/instance
I20260924 19:00:35.119545 388 external_mini_cluster.cc:1428] Setting key 20776601d213beb13c75a3076f9bb91d
Sep 24 19:00:35 dist-test-slave-9fn1 krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:35.126448 6211 heartbeater.cc:503] Master 127.0.97.62:45059 was elected leader, sending a full tablet report...
I20260924 19:00:35.132369 6213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:35.114912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55861 (local address 127.0.97.62:45059)
0924 19:00:35.115101 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:35.115108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:35.116238 (+ 1130us) server_negotiation.cc:407] Connection header received
0924 19:00:35.117937 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:35.117941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:35.118003 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:35.118090 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:35.120015 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:35.120989 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:35.122868 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:35.123240 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:35.123334 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:35.126598 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:35.126603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:35.126606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:35.126644 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:35.129409 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:35.130118 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:35.130128 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:35.130131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:35.130294 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:35.130722 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:35.130728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:35.130731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:35.130992 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:35.131161 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:35.131818 (+ 657us) server_negotiation.cc:299] Negotiation successful
0924 19:00:35.132039 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:00:35.133391 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:35.115430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45059 (local address 127.0.97.3:55861)
0924 19:00:35.116095 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:35.116131 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:35.117706 (+ 1575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:35.118277 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:35.118284 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:35.119017 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:35.119800 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:35.119826 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:35.121134 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:35.121140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:35.122732 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:35.122744 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:35.123419 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:35.123423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:35.123615 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:35.124216 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:35.124236 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:35.126400 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:35.129561 (+ 3161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:35.129572 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:35.129588 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:35.129991 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:35.130409 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:35.130414 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:35.130418 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:35.130602 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:35.131135 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:35.131142 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:35.131449 (+ 307us) client_negotiation.cc:769] Sending connection context
0924 19:00:35.131705 (+ 256us) client_negotiation.cc:240] Negotiation successful
0924 19:00:35.132206 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":268,"threads_started":1}
I20260924 19:00:35.134822 6478 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45059
I20260924 19:00:35.135058 6478 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:35.135669 6478 heartbeater.cc:511] Master 127.0.97.62:45059 requested a full tablet report, sending...
I20260924 19:00:35.136829 6022 ts_manager.cc:194] Registered new tserver with Master: cb0a670fdb814004844ec46ec2a070cc (127.0.97.3:44923)
I20260924 19:00:35.137666 6022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55861
I20260924 19:00:35.147097 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/test-admin.keytab.
Sep 24 19:00:35 dist-test-slave-9fn1 krb5kdc[5977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:35.172750 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 19:00:35.178489 6485 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:00:35.175455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51560 (local address 127.0.97.1:34435)
0924 19:00:35.176545 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:35.176557 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:35.176595 (+ 38us) server_negotiation.cc:407] Connection header received
0924 19:00:35.176679 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:35.176691 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:35.176971 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:35.177112 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:35.177991 (+ 879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51560: BlockingRecv error: recv got EOF from 127.0.0.1:51560 (error 108)
Metrics: {"server-negotiator.queue_time_us":879,"thread_start_us":615,"threads_started":1}
W20260924 19:00:35.178473 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/krb5cc' of type FILE
Sep 24 19:00:35 dist-test-slave-9fn1 krb5kdc[5977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:35.180053 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 19:00:35 dist-test-slave-9fn1 krb5kdc[5977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:35.181402 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 19:00:35 dist-test-slave-9fn1 krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 19:00:35.195770 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:35.183076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51576 (local address 127.0.97.1:34435)
0924 19:00:35.183265 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:35.183269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:35.183371 (+ 102us) server_negotiation.cc:407] Connection header received
0924 19:00:35.183498 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:35.183502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:35.183556 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:35.183636 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:35.184562 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:35.185576 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:35.186731 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:35.187265 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:35.187356 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:35.190673 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:35.190683 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:35.190697 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:35.190739 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:35.193281 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:35.193914 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:35.193921 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:35.193923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:35.194065 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:35.194388 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:35.194391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:35.194393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:35.194740 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:35.194954 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:35.195235 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 19:00:35.195483 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260924 19:00:35.201324 6485 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:00:35.199610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51578 (local address 127.0.97.1:34435)
0924 19:00:35.199807 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:35.199812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:35.199993 (+ 181us) server_negotiation.cc:407] Connection header received
0924 19:00:35.200177 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:35.200181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:35.200246 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:35.200344 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:35.201088 (+ 744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51578: BlockingRecv error: recv got EOF from 127.0.0.1:51578 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 19:00:35.201537 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/krb5cc' of type FILE
Sep 24 19:00:35 dist-test-slave-9fn1 krb5kdc[5977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:35.202973 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 19:00:35 dist-test-slave-9fn1 krb5kdc[5977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:35.204161 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 19:00:35 dist-test-slave-9fn1 krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 19:00:35.216960 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:35.205424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51594 (local address 127.0.97.1:34435)
0924 19:00:35.205676 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:35.205682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:35.205969 (+ 287us) server_negotiation.cc:407] Connection header received
0924 19:00:35.206089 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:35.206093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:35.206146 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:35.206223 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:35.207065 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:35.207735 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:35.208828 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:35.209249 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:35.209337 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:35.212626 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:35.212631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:35.212634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:35.212662 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:35.214983 (+ 2321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:35.215536 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:35.215539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:35.215541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:35.215654 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:35.215966 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:35.215969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:35.215971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:35.216201 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:35.216383 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:35.216589 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 19:00:35.216747 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 19:00:35.220798 6485 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:00:35.219299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51604 (local address 127.0.97.1:34435)
0924 19:00:35.219461 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:35.219464 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:35.219806 (+ 342us) server_negotiation.cc:407] Connection header received
0924 19:00:35.219911 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:35.219914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:35.219964 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:35.220037 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:35.220561 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51604: BlockingRecv error: recv got EOF from 127.0.0.1:51604 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 19:00:35.220992 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276386554690-388-0/krb5kdc/krb5cc' of type FILE
Sep 24 19:00:35 dist-test-slave-9fn1 krb5kdc[5977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:35.222416 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 19:00:35 dist-test-slave-9fn1 krb5kdc[5977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:35.223614 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 19:00:35 dist-test-slave-9fn1 krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 19:00:35.236284 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:35.224781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51618 (local address 127.0.97.1:34435)
0924 19:00:35.224929 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:35.224933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:35.225288 (+ 355us) server_negotiation.cc:407] Connection header received
0924 19:00:35.225387 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:35.225391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:35.225444 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:35.225578 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:35.226456 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:35.227138 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:35.228254 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:35.228635 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:35.228721 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:35.232002 (+ 3281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:35.232006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:35.232010 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:35.232036 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:35.234344 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:35.234888 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:35.234891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:35.234893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:35.234997 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:35.235282 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:35.235285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:35.235286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:35.235514 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:35.235738 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:35.235898 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 19:00:35.236039 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 19:00:35.237843 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 6083
I20260924 19:00:35.248871 6485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:35.238492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51620 (local address 127.0.97.1:34435)
0924 19:00:35.238625 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:35.238629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:35.238999 (+ 370us) server_negotiation.cc:407] Connection header received
0924 19:00:35.239116 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:35.239119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:35.239167 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:35.239239 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:35.240049 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:35.240677 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:35.241886 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:35.242283 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:35.242388 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:35.244328 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:35.244333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:35.244336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:35.244363 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:35.246892 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:35.247408 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:35.247412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:35.247414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:35.247528 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:35.247812 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:35.247815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:35.247817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:35.248050 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:35.248244 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:35.248488 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 19:00:35.248635 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 19:00:35.249395 6206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:35.334561 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 6083
I20260924 19:00:35.350620 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 6217
Sep 24 19:00:35 dist-test-slave-9fn1 krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 19:00:35.365614 6498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:35.351484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55288 (local address 127.0.97.2:42311)
0924 19:00:35.352104 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:35.352111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:35.352136 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:00:35.352204 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:35.352211 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:35.352404 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:35.352536 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:35.353396 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:35.354473 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:35.355713 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:35.356324 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:35.356419 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:35.360048 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:35.360058 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:35.360072 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:35.360102 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:35.362918 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:35.363587 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:35.363595 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:35.363598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:35.363715 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:35.364082 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:35.364088 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:35.364091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:35.364411 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:35.364712 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:35.365018 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 19:00:35.365244 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":353,"threads_started":1}
I20260924 19:00:35.366823 6340 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:35.448402 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 6217
I20260924 19:00:35.464205 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 6350
Sep 24 19:00:35 dist-test-slave-9fn1 krb5kdc[5977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 19:00:35.479040 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:35.465109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60926 (local address 127.0.97.3:44923)
0924 19:00:35.465723 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:35.465730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:35.465759 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:00:35.465816 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:35.465823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:35.466032 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:35.466155 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:35.467099 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:35.468075 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:35.469276 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:35.469901 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:35.470009 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:35.473781 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:35.473792 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:35.473807 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:35.473843 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:35.476571 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:35.477141 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:35.477148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:35.477151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:35.477259 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:35.477726 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:35.477730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:35.477732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:35.478053 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:35.478285 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:35.478490 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 19:00:35.478740 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":266,"threads_started":1}
I20260924 19:00:35.480254 6473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:35.561906 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 6350
I20260924 19:00:35.577471 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 5990
I20260924 19:00:35.578665 6052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:35.677399 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 5990
2026-09-24T19:00:35Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3854 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:00:35 dist-test-slave-9fn1 krb5kdc[6505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:00:35 dist-test-slave-9fn1 krb5kdc[6505](info): set up 2 sockets
Sep 24 19:00:35 dist-test-slave-9fn1 krb5kdc[6505](info): commencing operation
krb5kdc: starting...
W20260924 19:00:36.758219 388 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:00:36 dist-test-slave-9fn1 krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:00:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:00:36Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260924 19:00:36.965719 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46537
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46537
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:37.279865 6518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:37.280141 6518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:37.280320 6518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:37.287209 6518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:00:37.287324 6518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:37.287408 6518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:37.287494 6518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:00:37.287560 6518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:00:37.296692 6518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:37.297933 6518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:37.299158 6518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:37.310442 6526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:37.310523 6524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:37.310442 6523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:37.311573 6518 server_base.cc:1034] running on GCE node
I20260924 19:00:37.312275 6518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:37.313491 6518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:37.314740 6518 hybrid_clock.cc:648] HybridClock initialized: now 1790276437314706 us; error 24 us; skew 500 ppm
I20260924 19:00:37.315083 6518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:37 dist-test-slave-9fn1 krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276437, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:37.318001 6518 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260924 19:00:37.319403 6518 webserver.cc:460] Webserver started at http://127.0.97.62:41423/ using document root <none> and password file <none>
I20260924 19:00:37.320061 6518 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:37.320185 6518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:37.320478 6518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:37.323438 6518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "8f30647f1f9c43cf929f429a1e49e814"
format_stamp: "Formatted at 2026-09-24 19:00:37 on dist-test-slave-9fn1"
server_key: "ac39e2c48400cd7aea6f3734e6e41b4c"
server_key_iv: "afe92145ef7f73c6d125d6abfb78c8dc"
server_key_version: "encryptionkey@0"
I20260924 19:00:37.324179 6518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "8f30647f1f9c43cf929f429a1e49e814"
format_stamp: "Formatted at 2026-09-24 19:00:37 on dist-test-slave-9fn1"
server_key: "ac39e2c48400cd7aea6f3734e6e41b4c"
server_key_iv: "afe92145ef7f73c6d125d6abfb78c8dc"
server_key_version: "encryptionkey@0"
I20260924 19:00:37.329021 6518 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 19:00:37.332281 6533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:37.333639 6518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 19:00:37.333812 6518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "8f30647f1f9c43cf929f429a1e49e814"
format_stamp: "Formatted at 2026-09-24 19:00:37 on dist-test-slave-9fn1"
server_key: "ac39e2c48400cd7aea6f3734e6e41b4c"
server_key_iv: "afe92145ef7f73c6d125d6abfb78c8dc"
server_key_version: "encryptionkey@0"
I20260924 19:00:37.334014 6518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:37.350989 6518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:37.354691 6518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:37.354951 6518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:37.372462 6518 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46537
I20260924 19:00:37.372498 6584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46537 every 8 connection(s)
I20260924 19:00:37.373858 6518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:00:37.377393 6585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:37.383172 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 6518
I20260924 19:00:37.383365 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:00:37.384038 388 external_mini_cluster.cc:1428] Setting key 8613c8eeae2ae750c0451d1eccce3166
I20260924 19:00:37.388690 6585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814: Bootstrap starting.
I20260924 19:00:37.392611 6585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:37.393905 6585 log.cc:824] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:37.397037 6585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814: No bootstrap required, opened a new log
Sep 24 19:00:37 dist-test-slave-9fn1 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 19:00:37.406946 6585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } }
I20260924 19:00:37.407373 6585 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:37.407488 6585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f30647f1f9c43cf929f429a1e49e814, State: Initialized, Role: FOLLOWER
I20260924 19:00:37.408174 6585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } }
I20260924 19:00:37.408407 6588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:37.386121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33894 (local address 127.0.97.62:46537)
0924 19:00:37.386887 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:37.386902 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:37.386948 (+ 46us) server_negotiation.cc:407] Connection header received
0924 19:00:37.389194 (+ 2246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:37.389227 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:37.389774 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:37.390207 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:37.391445 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:37.393441 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:37.395076 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:37.396067 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:37.396252 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:37.400021 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:37.400031 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:37.400050 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:37.400095 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:37.404066 (+ 3971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:37.404671 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:37.404680 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:37.404690 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:37.404904 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:37.405241 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:37.405247 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:37.405251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:37.405977 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:37.406241 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:37.406577 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 19:00:37.407004 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":317,"threads_started":1}
I20260924 19:00:37.408445 6585 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:00:37.408569 6585 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:00:37.408754 6585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:37.411398 6585 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } }
I20260924 19:00:37.411914 6585 leader_election.cc:302] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f30647f1f9c43cf929f429a1e49e814; no voters:
I20260924 19:00:37.412454 6585 leader_election.cc:288] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:00:37.412606 6590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:00:37.413259 6590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [term 1 LEADER]: Becoming Leader. State: Replica: 8f30647f1f9c43cf929f429a1e49e814, State: Running, Role: LEADER
I20260924 19:00:37.413808 6590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } }
I20260924 19:00:37.414469 6585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:00:37.417238 6591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f30647f1f9c43cf929f429a1e49e814" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } }
I20260924 19:00:37.417217 6592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f30647f1f9c43cf929f429a1e49e814. Latest consensus state: current_term: 1 leader_uuid: "8f30647f1f9c43cf929f429a1e49e814" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } }
I20260924 19:00:37.417640 6591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:37.417716 6592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:37.420922 6596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:00:37.426285 6596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:00:37.435338 6596 catalog_manager.cc:1347] Generated new cluster ID: d0f1f7a463e442e28820eee13444c245
I20260924 19:00:37.435470 6596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:00:37.452673 6596 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:00:37.454840 6596 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:00:37.468983 6596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814: Generated new TSK 0
I20260924 19:00:37.470115 6596 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260924 19:00:37.534900 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46537
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:37.862110 6611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:37.862428 6611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:37.862596 6611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:37.869844 6611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:37.869999 6611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:37.870174 6611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:00:37.880009 6611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:37.881258 6611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:37.882550 6611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:37.895123 6617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:37.895272 6616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:37.895212 6619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:37.896076 6611 server_base.cc:1034] running on GCE node
I20260924 19:00:37.896622 6611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:37.897476 6611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:37.898747 6611 hybrid_clock.cc:648] HybridClock initialized: now 1790276437898685 us; error 58 us; skew 500 ppm
I20260924 19:00:37.899014 6611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:37 dist-test-slave-9fn1 krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276437, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:37.902051 6611 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260924 19:00:37.903514 6611 webserver.cc:460] Webserver started at http://127.0.97.1:37121/ using document root <none> and password file <none>
I20260924 19:00:37.904196 6611 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:37.904323 6611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:37.904616 6611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:37.907516 6611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/data/instance:
uuid: "31b40f242c874e81a6ab82ce8cb210c7"
format_stamp: "Formatted at 2026-09-24 19:00:37 on dist-test-slave-9fn1"
server_key: "f76bb72716da4356a5119834c4d72a18"
server_key_iv: "3e45165b206e313a38dda5a03539ec9f"
server_key_version: "encryptionkey@0"
I20260924 19:00:37.908282 6611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/wal/instance:
uuid: "31b40f242c874e81a6ab82ce8cb210c7"
format_stamp: "Formatted at 2026-09-24 19:00:37 on dist-test-slave-9fn1"
server_key: "f76bb72716da4356a5119834c4d72a18"
server_key_iv: "3e45165b206e313a38dda5a03539ec9f"
server_key_version: "encryptionkey@0"
I20260924 19:00:37.913228 6611 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 19:00:37.917253 6626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:37.918756 6611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:00:37.918936 6611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "31b40f242c874e81a6ab82ce8cb210c7"
format_stamp: "Formatted at 2026-09-24 19:00:37 on dist-test-slave-9fn1"
server_key: "f76bb72716da4356a5119834c4d72a18"
server_key_iv: "3e45165b206e313a38dda5a03539ec9f"
server_key_version: "encryptionkey@0"
I20260924 19:00:37.919134 6611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:37.940303 6611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:37.944610 6611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:37.944900 6611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:37.945917 6611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:37.947190 6611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:37.947268 6611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:37.947439 6611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:37.947499 6611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:37.978243 6611 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33575
I20260924 19:00:37.978322 6738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33575 every 8 connection(s)
I20260924 19:00:37.979467 6611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:00:37.987061 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 6611
I20260924 19:00:37.987255 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/wal/instance
I20260924 19:00:37.987915 388 external_mini_cluster.cc:1428] Setting key dd419d0d3cf0697c8f3bb21eeefd0032
Sep 24 19:00:37 dist-test-slave-9fn1 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276437, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 19:00:37.999011 6741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:37.981825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51107 (local address 127.0.97.62:46537)
0924 19:00:37.982221 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:37.982226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:37.983249 (+ 1023us) server_negotiation.cc:407] Connection header received
0924 19:00:37.984987 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:37.984992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:37.985055 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:37.985151 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:37.986624 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:37.987421 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:37.989095 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:37.989660 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:37.989790 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:37.993350 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:37.993355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:37.993358 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:37.993389 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:37.995955 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:37.996696 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:37.996708 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:37.996711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:37.996824 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:37.997254 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:37.997261 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:37.997264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:37.997592 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:37.997741 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:37.998431 (+ 690us) server_negotiation.cc:299] Negotiation successful
0924 19:00:37.998656 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":187,"threads_started":1}
I20260924 19:00:37.999992 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:37.982421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.97.1:51107)
0924 19:00:37.983084 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:37.983126 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:37.984749 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:37.985382 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:37.985389 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:37.985932 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:37.986469 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:37.986485 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:37.987557 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:37.987565 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:37.988942 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:37.988953 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:37.989877 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:37.989883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:37.990110 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:37.990788 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:37.990814 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:37.993128 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:37.996114 (+ 2986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:37.996123 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:37.996140 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:37.996559 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:37.996939 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:37.996944 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:37.996948 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:37.997143 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:37.997715 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:37.997722 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:37.998067 (+ 345us) client_negotiation.cc:769] Sending connection context
0924 19:00:37.998341 (+ 274us) client_negotiation.cc:240] Negotiation successful
0924 19:00:37.998880 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":259,"threads_started":1}
I20260924 19:00:38.001565 6739 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46537
I20260924 19:00:38.001875 6739 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:38.002569 6739 heartbeater.cc:511] Master 127.0.97.62:46537 requested a full tablet report, sending...
I20260924 19:00:38.004977 6550 ts_manager.cc:194] Registered new tserver with Master: 31b40f242c874e81a6ab82ce8cb210c7 (127.0.97.1:33575)
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 19:00:38.007256 6550 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260924 19:00:38.032649 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46537
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:38.347317 6745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:38.347599 6745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:38.347760 6745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:38.355180 6745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:38.355284 6745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:38.355485 6745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:00:38.365198 6745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:38.366575 6745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:38.367887 6745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:38.381258 6751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:38.381295 6753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:38.386147 6750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:38.386794 6745 server_base.cc:1034] running on GCE node
I20260924 19:00:38.387424 6745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:38.388319 6745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:38.389604 6745 hybrid_clock.cc:648] HybridClock initialized: now 1790276438389533 us; error 55 us; skew 500 ppm
I20260924 19:00:38.389907 6745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:38 dist-test-slave-9fn1 krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276438, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:38.392777 6745 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260924 19:00:38.394251 6745 webserver.cc:460] Webserver started at http://127.0.97.2:44301/ using document root <none> and password file <none>
I20260924 19:00:38.394892 6745 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:38.395030 6745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:38.395316 6745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:38.398108 6745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/data/instance:
uuid: "56fd3560adc24fa5aa71678e700f9836"
format_stamp: "Formatted at 2026-09-24 19:00:38 on dist-test-slave-9fn1"
server_key: "1d5ce75eec9c72a6cd432501ddd9f6b6"
server_key_iv: "29af93175fa4ae99d21bee60b4eccacf"
server_key_version: "encryptionkey@0"
I20260924 19:00:38.398842 6745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/wal/instance:
uuid: "56fd3560adc24fa5aa71678e700f9836"
format_stamp: "Formatted at 2026-09-24 19:00:38 on dist-test-slave-9fn1"
server_key: "1d5ce75eec9c72a6cd432501ddd9f6b6"
server_key_iv: "29af93175fa4ae99d21bee60b4eccacf"
server_key_version: "encryptionkey@0"
I20260924 19:00:38.403620 6745 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 19:00:38.406910 6760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:38.408092 6745 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 19:00:38.408245 6745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "56fd3560adc24fa5aa71678e700f9836"
format_stamp: "Formatted at 2026-09-24 19:00:38 on dist-test-slave-9fn1"
server_key: "1d5ce75eec9c72a6cd432501ddd9f6b6"
server_key_iv: "29af93175fa4ae99d21bee60b4eccacf"
server_key_version: "encryptionkey@0"
I20260924 19:00:38.408425 6745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:38.427819 6745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:38.431651 6745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:38.431957 6745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:38.432897 6745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:38.434166 6745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:38.434271 6745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:38.434419 6745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:38.434490 6745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:38.466421 6745 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44407
I20260924 19:00:38.466506 6872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44407 every 8 connection(s)
I20260924 19:00:38.467639 6745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:00:38.471825 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 6745
I20260924 19:00:38.472021 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/wal/instance
I20260924 19:00:38.472437 388 external_mini_cluster.cc:1428] Setting key 3776cd74c6b6588ce7690f2bf7f3dc9c
Sep 24 19:00:38 dist-test-slave-9fn1 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276438, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 19:00:38.487092 6741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:38.470254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53447 (local address 127.0.97.62:46537)
0924 19:00:38.470443 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:38.470449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:38.471625 (+ 1176us) server_negotiation.cc:407] Connection header received
0924 19:00:38.473637 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:38.473642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:38.473725 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:38.473851 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:38.475434 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:38.476108 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:38.477707 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:38.478131 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:38.478259 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:38.481494 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:38.481551 (+ 57us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:38.481556 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:38.481596 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:38.484093 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:38.484780 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:38.484787 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:38.484790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:38.484914 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:38.485334 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:38.485340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:38.485343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:38.485691 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:38.485867 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:38.486506 (+ 639us) server_negotiation.cc:299] Negotiation successful
0924 19:00:38.486722 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:00:38.488058 6875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:38.470808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.97.2:53447)
0924 19:00:38.471468 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:38.471510 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:38.473312 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:38.474112 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:38.474124 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:38.474741 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:38.475269 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:38.475286 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:38.476254 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:38.476261 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:38.477566 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:38.477578 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:38.478354 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:38.478362 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:38.478583 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:38.479249 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:38.479273 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:38.481281 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:38.484244 (+ 2963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:38.484252 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:38.484267 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:38.484662 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:38.485023 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:38.485029 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:38.485035 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:38.485222 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:38.485839 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:38.485847 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:38.486165 (+ 318us) client_negotiation.cc:769] Sending connection context
0924 19:00:38.486422 (+ 257us) client_negotiation.cc:240] Negotiation successful
0924 19:00:38.486940 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":254,"threads_started":1}
I20260924 19:00:38.489470 6873 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46537
I20260924 19:00:38.489810 6873 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 19:00:38.490538 6873 heartbeater.cc:511] Master 127.0.97.62:46537 requested a full tablet report, sending...
I20260924 19:00:38.491995 6550 ts_manager.cc:194] Registered new tserver with Master: 56fd3560adc24fa5aa71678e700f9836 (127.0.97.2:44407)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
I20260924 19:00:38.492990 6550 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:53447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260924 19:00:38.517081 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46537
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:38.827971 6878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:38.828226 6878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:38.828415 6878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:38.835464 6878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:38.835594 6878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:38.835778 6878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:00:38.845199 6878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:38.846454 6878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:38.847661 6878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:38.860136 6883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:38.860208 6886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:38.860160 6884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:38.860996 6878 server_base.cc:1034] running on GCE node
I20260924 19:00:38.861595 6878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:38.862424 6878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:38.863636 6878 hybrid_clock.cc:648] HybridClock initialized: now 1790276438863574 us; error 54 us; skew 500 ppm
I20260924 19:00:38.863888 6878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:38 dist-test-slave-9fn1 krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276438, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:38.866869 6878 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260924 19:00:38.868148 6878 webserver.cc:460] Webserver started at http://127.0.97.3:36465/ using document root <none> and password file <none>
I20260924 19:00:38.868762 6878 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:38.868858 6878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:38.869094 6878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:38.871865 6878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/data/instance:
uuid: "83c576b1596f4264a2d7ce70c7ec7c4a"
format_stamp: "Formatted at 2026-09-24 19:00:38 on dist-test-slave-9fn1"
server_key: "31f263a4edacaac629d28c100cbe4887"
server_key_iv: "57e06f296b1cac9de781eb5dca61c64a"
server_key_version: "encryptionkey@0"
I20260924 19:00:38.872561 6878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/wal/instance:
uuid: "83c576b1596f4264a2d7ce70c7ec7c4a"
format_stamp: "Formatted at 2026-09-24 19:00:38 on dist-test-slave-9fn1"
server_key: "31f263a4edacaac629d28c100cbe4887"
server_key_iv: "57e06f296b1cac9de781eb5dca61c64a"
server_key_version: "encryptionkey@0"
I20260924 19:00:38.877425 6878 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:00:38.880728 6893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:38.882095 6878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 19:00:38.882222 6878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "83c576b1596f4264a2d7ce70c7ec7c4a"
format_stamp: "Formatted at 2026-09-24 19:00:38 on dist-test-slave-9fn1"
server_key: "31f263a4edacaac629d28c100cbe4887"
server_key_iv: "57e06f296b1cac9de781eb5dca61c64a"
server_key_version: "encryptionkey@0"
I20260924 19:00:38.882385 6878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:38.914033 6878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:38.918128 6878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:38.918355 6878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:38.919273 6878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:38.920502 6878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:38.920572 6878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:38.920656 6878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:38.920698 6878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:38.952980 6878 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44055
I20260924 19:00:38.953035 7005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44055 every 8 connection(s)
I20260924 19:00:38.954282 6878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:00:38.956705 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 6878
I20260924 19:00:38.956893 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/wal/instance
I20260924 19:00:38.957459 388 external_mini_cluster.cc:1428] Setting key 1bd8498ec78680ec03f8a63a269462ad
Sep 24 19:00:38 dist-test-slave-9fn1 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276438, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 19:00:38.973327 6741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:38.956843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50231 (local address 127.0.97.62:46537)
0924 19:00:38.957075 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:38.957080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:38.958551 (+ 1471us) server_negotiation.cc:407] Connection header received
0924 19:00:38.960556 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:38.960561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:38.960647 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:38.960748 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:38.962205 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:38.962938 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:38.964417 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:38.964794 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:38.964928 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:38.968129 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:38.968134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:38.968138 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:38.968165 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:38.970458 (+ 2293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:38.971224 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:38.971230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:38.971233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:38.971404 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:38.971887 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:38.971892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:38.971895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:38.972160 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:38.972317 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:38.972941 (+ 624us) server_negotiation.cc:299] Negotiation successful
0924 19:00:38.973093 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260924 19:00:38.974201 7008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:38.957542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.97.3:50231)
0924 19:00:38.958398 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:38.958438 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:38.960315 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:38.960978 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:38.960986 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:38.961447 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:38.962049 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:38.962065 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:38.963090 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:38.963094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:38.964267 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:38.964278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:38.965020 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:38.965027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:38.965238 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:38.965875 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:38.965899 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:38.967915 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:38.970597 (+ 2682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:38.970604 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:38.970618 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:38.971099 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:38.971530 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:38.971537 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:38.971540 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:38.971788 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:38.972315 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:38.972323 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:38.972605 (+ 282us) client_negotiation.cc:769] Sending connection context
0924 19:00:38.972830 (+ 225us) client_negotiation.cc:240] Negotiation successful
0924 19:00:38.973162 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":338,"threads_started":1}
I20260924 19:00:38.975451 7006 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46537
I20260924 19:00:38.975747 7006 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:38.976388 7006 heartbeater.cc:511] Master 127.0.97.62:46537 requested a full tablet report, sending...
I20260924 19:00:38.977706 6550 ts_manager.cc:194] Registered new tserver with Master: 83c576b1596f4264a2d7ce70c7ec7c4a (127.0.97.3:44055)
I20260924 19:00:38.978492 6550 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50231
I20260924 19:00:38.984392 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
I20260924 19:00:39.010720 6739 heartbeater.cc:503] Master 127.0.97.62:46537 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260924 19:00:39.031235 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37383
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46537,127.0.97.61:37383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:39.343523 7011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:39.343803 7011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:39.343889 7011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:39.350936 7011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:00:39.351043 7011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:39.351089 7011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:39.351146 7011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:00:39.351182 7011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:00:39.360618 7011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46537,127.0.97.61:37383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:37383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
W20260924 19:00:39.360908 7011 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:46537,127.0.97.61:37383'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 19:00:39.361867 7011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:39.363018 7011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:39.373986 7017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:39.374053 7019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:39.374574 7016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:39.375104 7011 server_base.cc:1034] running on GCE node
I20260924 19:00:39.375957 7011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:39.377646 7011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:39.378895 7011 hybrid_clock.cc:648] HybridClock initialized: now 1790276439378812 us; error 44 us; skew 500 ppm
I20260924 19:00:39.379246 7011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:39 dist-test-slave-9fn1 krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276439, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:39.382771 7011 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260924 19:00:39.384562 7011 webserver.cc:460] Webserver started at http://127.0.97.61:32857/ using document root <none> and password file <none>
I20260924 19:00:39.385319 7011 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:39.385449 7011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:39.385792 7011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:39.388731 7011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-1/data/instance:
uuid: "07bd526e5ab649bb873f289aee41b508"
format_stamp: "Formatted at 2026-09-24 19:00:39 on dist-test-slave-9fn1"
server_key: "257573449a0b38acaeae8eaf3b25da5a"
server_key_iv: "fd4ddab6aa9a7c5b0b71f25bb5b5e7ce"
server_key_version: "encryptionkey@0"
I20260924 19:00:39.389493 7011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-1/wal/instance:
uuid: "07bd526e5ab649bb873f289aee41b508"
format_stamp: "Formatted at 2026-09-24 19:00:39 on dist-test-slave-9fn1"
server_key: "257573449a0b38acaeae8eaf3b25da5a"
server_key_iv: "fd4ddab6aa9a7c5b0b71f25bb5b5e7ce"
server_key_version: "encryptionkey@0"
I20260924 19:00:39.394563 7011 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 19:00:39.397819 7026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:39.399076 7011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 19:00:39.399243 7011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-1/wal
uuid: "07bd526e5ab649bb873f289aee41b508"
format_stamp: "Formatted at 2026-09-24 19:00:39 on dist-test-slave-9fn1"
server_key: "257573449a0b38acaeae8eaf3b25da5a"
server_key_iv: "fd4ddab6aa9a7c5b0b71f25bb5b5e7ce"
server_key_version: "encryptionkey@0"
I20260924 19:00:39.399441 7011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:39.445017 7011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:39.448922 7011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:39.449193 7011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:39.467061 7011 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:37383
I20260924 19:00:39.467078 7077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:37383 every 8 connection(s)
I20260924 19:00:39.468307 7011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-1/data/info.pb
I20260924 19:00:39.469631 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 7011
I20260924 19:00:39.472931 7078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:39.479570 7078 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:00:39 dist-test-slave-9fn1 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 24 19:00:39 dist-test-slave-9fn1 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276439, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 19:00:39.492805 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:39.471856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45002 (local address 127.0.97.61:37383)
0924 19:00:39.473058 (+ 1202us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:39.473075 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:39.473124 (+ 49us) server_negotiation.cc:407] Connection header received
0924 19:00:39.474757 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:39.474787 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:39.475184 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:39.475580 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:39.476948 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:39.478324 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:39.479520 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:39.480175 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:39.480355 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:39.485141 (+ 4786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:39.485151 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:39.485164 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:39.485202 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:39.488431 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:39.489176 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:39.489181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:39.489187 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:39.489325 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:39.489875 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:39.489883 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:39.489889 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:39.490520 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:39.490734 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:39.491180 (+ 446us) server_negotiation.cc:299] Negotiation successful
0924 19:00:39.491522 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":980,"thread_start_us":769,"threads_started":1}
I20260924 19:00:39.496141 6873 heartbeater.cc:503] Master 127.0.97.62:46537 was elected leader, sending a full tablet report...
I20260924 19:00:39.497430 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:39.481911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33910 (local address 127.0.97.62:46537)
0924 19:00:39.482996 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:39.483001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:39.483015 (+ 14us) server_negotiation.cc:407] Connection header received
0924 19:00:39.483060 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:39.483064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:39.483117 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:39.483239 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:39.484574 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:39.485521 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:39.487065 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:39.487462 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:39.487591 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:39.491423 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:39.491429 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:39.491435 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:39.491474 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:39.494531 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:39.495351 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:39.495358 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:39.495364 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:39.495501 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:39.495897 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:39.495900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:39.495902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:39.496176 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:39.496324 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:39.496855 (+ 531us) server_negotiation.cc:299] Negotiation successful
0924 19:00:39.497057 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":994,"thread_start_us":902,"threads_started":1}
I20260924 19:00:39.497462 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:39.481830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.0.1:33910)
0924 19:00:39.482465 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:39.482503 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:39.482636 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:39.483290 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:39.483299 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:39.484120 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:39.484430 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:39.484441 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:39.485668 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:39.485678 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:39.486934 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:39.486946 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:39.487658 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:39.487665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:39.487824 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:39.488387 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:39.488444 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:39.491240 (+ 2796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:39.494655 (+ 3415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:39.494663 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:39.494669 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:39.495229 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:39.495617 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:39.495623 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:39.495626 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:39.495789 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:39.496302 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:39.496309 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:39.496508 (+ 199us) client_negotiation.cc:769] Sending connection context
0924 19:00:39.496749 (+ 241us) client_negotiation.cc:240] Negotiation successful
0924 19:00:39.497045 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"spinlock_wait_cycles":12672,"thread_start_us":287,"threads_started":1}
I20260924 19:00:39.498808 7078 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:00:39 dist-test-slave-9fn1 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276439, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260924 19:00:39.510202 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:39.499482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37383 (local address 127.0.0.1:45014)
0924 19:00:39.499672 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:39.499694 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:39.499770 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:39.500015 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:39.500018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:39.500300 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:39.500426 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:39.500433 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:39.501303 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:39.501306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:39.502185 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:39.502195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:39.502890 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:39.502894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:39.503035 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:39.503480 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:39.503495 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:39.505427 (+ 1932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:39.508173 (+ 2746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:39.508177 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:39.508179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:39.508530 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:39.508861 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:39.508864 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:39.508866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:39.508975 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:39.509495 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:39.509546 (+ 51us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:39.509673 (+ 127us) client_negotiation.cc:769] Sending connection context
0924 19:00:39.509780 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 19:00:39.509967 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 19:00:39.510447 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:39.499549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45014 (local address 127.0.97.61:37383)
0924 19:00:39.499699 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:39.499703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:39.499747 (+ 44us) server_negotiation.cc:407] Connection header received
0924 19:00:39.499847 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:39.499851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:39.499899 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:39.499974 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:39.500540 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:39.501205 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:39.502297 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:39.502769 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:39.502860 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:39.505614 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:39.505619 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:39.505622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:39.505647 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:39.508066 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:39.508621 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:39.508625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:39.508627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:39.508738 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:39.509073 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:39.509078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:39.509082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:39.509349 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:39.509567 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:39.509859 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 19:00:39.510081 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":50}
I20260924 19:00:39.515311 7078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508: Bootstrap starting.
I20260924 19:00:39.518456 7078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:39.519415 7078 log.cc:824] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:39.521955 7078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508: No bootstrap required, opened a new log
I20260924 19:00:39.528913 7078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } }
I20260924 19:00:39.529233 7078 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:39.529330 7078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07bd526e5ab649bb873f289aee41b508, State: Initialized, Role: FOLLOWER
I20260924 19:00:39.529958 7078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } }
I20260924 19:00:39.530725 7085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } } }
I20260924 19:00:39.531051 7085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:00:39.531709 7078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 19:00:39.539671 7096 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:00:39.539810 7096 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:00:39.542478 7011 master_runner.cc:428] Detected that this master 07bd526e5ab649bb873f289aee41b508 is joining an existing cluster
I20260924 19:00:39.542568 7011 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:37383
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
I20260924 19:00:39.575438 7103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:39.546467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37383 (local address 127.0.0.1:45028)
0924 19:00:39.559258 (+ 12791us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:39.559276 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:39.559397 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:39.559862 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:39.559870 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:39.560545 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:39.560870 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:39.560884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:39.562476 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:39.562484 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:39.564290 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:39.564308 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:39.565716 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:39.565724 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:39.565938 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:39.566574 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:39.566599 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:39.567704 (+ 1105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:39.571906 (+ 4202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:39.571909 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:39.571912 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:39.572335 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:39.572976 (+ 641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:39.572979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:39.572981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:39.573102 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:39.574267 (+ 1165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:39.574272 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:39.574415 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 19:00:39.574578 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 19:00:39.574913 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12656,"mutex_wait_us":29,"thread_start_us":12502,"threads_started":1}
I20260924 19:00:39.575606 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:39.546891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45028 (local address 127.0.97.61:37383)
0924 19:00:39.547143 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:39.547153 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:39.559374 (+ 12221us) server_negotiation.cc:407] Connection header received
0924 19:00:39.559512 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:39.559521 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:39.559601 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:39.559701 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:39.561022 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:39.562264 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:39.564442 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:39.565253 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:39.565381 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:39.567919 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:39.567928 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:39.567936 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:39.567980 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:39.571751 (+ 3771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:39.572489 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:39.572497 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:39.572500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:39.572610 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:39.573200 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:39.573208 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:39.573212 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:39.573668 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:39.573874 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:39.574712 (+ 838us) server_negotiation.cc:299] Negotiation successful
0924 19:00:39.575085 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":118,"server-negotiator.queue_time_us":62}
I20260924 19:00:39.576388 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:39.547303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33920 (local address 127.0.97.62:46537)
0924 19:00:39.547508 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:39.547517 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:39.556795 (+ 9278us) server_negotiation.cc:407] Connection header received
0924 19:00:39.557044 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:39.557051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:39.557192 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:39.557277 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:39.559568 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:39.560692 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:39.562856 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:39.563520 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:39.563617 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:39.566559 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:39.566565 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:39.566570 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:39.566608 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:39.571760 (+ 5152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:39.572729 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:39.572737 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:39.572744 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:39.572863 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:39.573341 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:39.573346 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:39.573349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:39.573673 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:39.573810 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:39.575795 (+ 1985us) server_negotiation.cc:299] Negotiation successful
0924 19:00:39.576035 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 19:00:39.577772 7102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:39.546372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.0.1:33920)
0924 19:00:39.556653 (+ 10281us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:39.556681 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:39.556862 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:39.557552 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:39.557557 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:39.558303 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:39.558709 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:39.558722 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:39.560888 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:39.560894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:39.562671 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:39.562691 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:39.563920 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:39.563926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:39.564139 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:39.564908 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:39.564938 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:39.566073 (+ 1135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:39.571935 (+ 5862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:39.571940 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:39.571944 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:39.572481 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:39.572982 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:39.572987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:39.572992 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:39.573160 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:39.573868 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:39.573873 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:39.574104 (+ 231us) client_negotiation.cc:769] Sending connection context
0924 19:00:39.576554 (+ 2450us) client_negotiation.cc:240] Negotiation successful
0924 19:00:39.577252 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10118,"mutex_wait_us":3,"thread_start_us":9964,"threads_started":1}
Sep 24 19:00:39 dist-test-slave-9fn1 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276437, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260924 19:00:39.603434 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:43355
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46537,127.0.97.61:37383,127.0.97.60:43355
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:00:39.604686 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:39.584268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45044 (local address 127.0.97.61:37383)
0924 19:00:39.585706 (+ 1438us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:39.585713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:39.586443 (+ 730us) server_negotiation.cc:407] Connection header received
0924 19:00:39.586828 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:39.586837 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:39.586971 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:39.587070 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:39.588465 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:39.589755 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:39.591658 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:39.592473 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:39.592622 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:39.598057 (+ 5435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:39.598085 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:39.598093 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:39.598136 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:39.601367 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:39.602226 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:39.602233 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:39.602237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:39.602411 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:39.602824 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:39.602829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:39.602833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:39.603260 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:39.603407 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:39.604075 (+ 668us) server_negotiation.cc:299] Negotiation successful
0924 19:00:39.604298 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1309}
I20260924 19:00:39.604876 7106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:39.584134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37383 (local address 127.0.0.1:45044)
0924 19:00:39.586288 (+ 2154us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:39.586340 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:39.586599 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:39.587119 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:39.587127 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:39.588018 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:39.588305 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:39.588321 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:39.589893 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:39.589907 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:39.591490 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:39.591505 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:39.592663 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:39.592671 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:39.592925 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:39.593806 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:39.593835 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:39.597860 (+ 4025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:39.601548 (+ 3688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:39.601559 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:39.601567 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:39.602093 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:39.602515 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:39.602520 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:39.602525 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:39.602692 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:39.603459 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:39.603489 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:39.603719 (+ 230us) client_negotiation.cc:769] Sending connection context
0924 19:00:39.603941 (+ 222us) client_negotiation.cc:240] Negotiation successful
0924 19:00:39.604320 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1903,"thread_start_us":1687,"threads_started":1}
I20260924 19:00:39.607101 6550 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:37383: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "07bd526e5ab649bb873f289aee41b508"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 37383
}
attrs {
promote: true
}
}
dest_uuid: "8f30647f1f9c43cf929f429a1e49e814"
cas_config_opid_index: -1
I20260924 19:00:39.608243 6550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: true } }
I20260924 19:00:39.622613 7109 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 07bd526e5ab649bb873f289aee41b508 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: true } } }
I20260924 19:00:39.624167 7053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:39.627630 7109 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:37383
I20260924 19:00:39.627975 7053 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [term 1 FOLLOWER]: Refusing update from remote peer 8f30647f1f9c43cf929f429a1e49e814: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 19:00:39.628259 7110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8f30647f1f9c43cf929f429a1e49e814" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: true } } }
I20260924 19:00:39.628564 7110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:39.628355 7011 master.cc:537] Master@127.0.97.61:37383 shutting down...
I20260924 19:00:39.633136 7109 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:00:39.637401 6534 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260924 19:00:39.641832 7011 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:00:39.642244 7011 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:39.642417 7011 tablet_replica.cc:330] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508: stopping tablet replica
W20260924 19:00:39.642452 6534 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 -> Peer 07bd526e5ab649bb873f289aee41b508 (127.0.97.61:37383): Couldn't send request to peer 07bd526e5ab649bb873f289aee41b508. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:00:39.648677 7011 master.cc:552] Master@127.0.97.61:37383 shutdown complete.
I20260924 19:00:39.648793 7011 master_runner.cc:305] Clearing existing system tablet
I20260924 19:00:39.649251 7011 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:39.649374 7011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:39.652221 7011 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260924 19:00:39.653909 7114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:39.654266 7011 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:39.654655 7011 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 19:00:39.654786 7011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-1/wal
uuid: "07bd526e5ab649bb873f289aee41b508"
format_stamp: "Formatted at 2026-09-24 19:00:39 on dist-test-slave-9fn1"
server_key: "257573449a0b38acaeae8eaf3b25da5a"
server_key_iv: "fd4ddab6aa9a7c5b0b71f25bb5b5e7ce"
server_key_version: "encryptionkey@0"
I20260924 19:00:39.659446 7011 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:39.666466 7011 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 19:00:39.666713 7011 log.cc:1196] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 19:00:39.667223 7011 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508: Deleting consensus metadata
I20260924 19:00:39.667514 7011 master_runner.cc:315] Copying system tablet from 127.0.97.62:46537
I20260924 19:00:39.670182 7011 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:46537
I20260924 19:00:39.690876 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:39.671268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33928 (local address 127.0.97.62:46537)
0924 19:00:39.671498 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:39.671507 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:39.676954 (+ 5447us) server_negotiation.cc:407] Connection header received
0924 19:00:39.677134 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:39.677140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:39.677252 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:39.677368 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:39.678434 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:39.679563 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:39.681350 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:39.681981 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:39.682110 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:39.684300 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:39.684307 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:39.684313 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:39.684352 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:39.687830 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:39.688655 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:39.688660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:39.688664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:39.688828 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:39.689271 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:39.689276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:39.689280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:39.689652 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:39.689800 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:39.690275 (+ 475us) server_negotiation.cc:299] Negotiation successful
0924 19:00:39.690497 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 19:00:39.690922 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:39.671048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.0.1:33928)
0924 19:00:39.676797 (+ 5749us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:39.676828 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:39.676999 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:39.677415 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:39.677420 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:39.677981 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:39.678262 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:39.678274 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:39.679725 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:39.679730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:39.681187 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:39.681202 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:39.682146 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:39.682153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:39.682323 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:39.683041 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:39.683068 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:39.684100 (+ 1032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:39.687984 (+ 3884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:39.687991 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:39.687996 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:39.688517 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:39.688966 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:39.688972 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:39.688979 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:39.689158 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:39.689788 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:39.689796 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:39.690004 (+ 208us) client_negotiation.cc:769] Sending connection context
0924 19:00:39.690163 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 19:00:39.690481 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5579,"thread_start_us":5479,"threads_started":1}
I20260924 19:00:39.692747 6570 tablet_copy_service.cc:140] P 8f30647f1f9c43cf929f429a1e49e814: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 07bd526e5ab649bb873f289aee41b508 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:33928)
I20260924 19:00:39.693056 6570 tablet_copy_service.cc:161] P 8f30647f1f9c43cf929f429a1e49e814: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 07bd526e5ab649bb873f289aee41b508 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:33928: session id = 07bd526e5ab649bb873f289aee41b508-00000000000000000000000000000000
I20260924 19:00:39.697674 6570 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 19:00:39.700002 7011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 19:00:39.707382 7011 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508: tablet copy: Starting download of 0 data blocks...
I20260924 19:00:39.707690 7011 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508: tablet copy: Starting download of 1 WAL segments...
I20260924 19:00:39.710539 7011 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 19:00:39.715126 6570 tablet_copy_service.cc:342] P 8f30647f1f9c43cf929f429a1e49e814: Request end of tablet copy session 07bd526e5ab649bb873f289aee41b508-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:33928
I20260924 19:00:39.715339 6570 tablet_copy_service.cc:433] P 8f30647f1f9c43cf929f429a1e49e814: ending tablet copy session 07bd526e5ab649bb873f289aee41b508-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 07bd526e5ab649bb873f289aee41b508
I20260924 19:00:39.726778 7011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:39.729455 7121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:39.734920 7122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:39.742066 7124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:39.742545 7011 server_base.cc:1034] running on GCE node
I20260924 19:00:39.742899 7011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:39.743602 7011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:39.744889 7011 hybrid_clock.cc:648] HybridClock initialized: now 1790276439744823 us; error 32 us; skew 500 ppm
I20260924 19:00:39.745304 7011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:39 dist-test-slave-9fn1 krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276439, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:39.747623 7011 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260924 19:00:39.760736 7011 webserver.cc:460] Webserver started at http://127.0.97.61:33637/ using document root <none> and password file <none>
I20260924 19:00:39.761205 7011 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:39.761363 7011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:39.764380 7011 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260924 19:00:39.766312 7131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:39.766793 7011 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:00:39.766984 7011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-1/wal
uuid: "07bd526e5ab649bb873f289aee41b508"
format_stamp: "Formatted at 2026-09-24 19:00:39 on dist-test-slave-9fn1"
server_key: "257573449a0b38acaeae8eaf3b25da5a"
server_key_iv: "fd4ddab6aa9a7c5b0b71f25bb5b5e7ce"
server_key_version: "encryptionkey@0"
I20260924 19:00:39.767207 7011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:39.814409 7011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:39.820122 7011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:39.846937 7182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:37383 every 8 connection(s)
I20260924 19:00:39.846967 7011 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:37383
I20260924 19:00:39.848099 7011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-1/data/info.pb
I20260924 19:00:39.853319 7183 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:00:39.857339 7183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508: Bootstrap starting.
I20260924 19:00:39.875506 7183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:00:39.876045 7183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508: Bootstrap complete.
I20260924 19:00:39.877097 7183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: true } }
I20260924 19:00:39.877367 7183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 07bd526e5ab649bb873f289aee41b508, State: Initialized, Role: LEARNER
I20260924 19:00:39.877638 7183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: true } }
I20260924 19:00:39.878458 7186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: true } } }
I20260924 19:00:39.878742 7183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:00:39.878958 7186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [sys.catalog]: This master's current role is: LEARNER
I20260924 19:00:39.889002 7197 catalog_manager.cc:1259] Loaded cluster ID: d0f1f7a463e442e28820eee13444c245
I20260924 19:00:39.889117 7197 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508: loading cluster ID for follower catalog manager: success
I20260924 19:00:39.892683 7197 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508: acquiring CA information for follower catalog manager: success
I20260924 19:00:39.895226 7197 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:00:39.980769 7006 heartbeater.cc:503] Master 127.0.97.62:46537 was elected leader, sending a full tablet report...
W20260924 19:00:40.012293 7107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:40.012665 7107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:40.012820 7107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:40.020963 7107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:00:40.021056 7107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:40.021175 7107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:40.021276 7107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:00:40.021356 7107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:00:40.032075 7107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46537,127.0.97.61:37383,127.0.97.60:43355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:43355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:40.033538 7107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:40.034895 7107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:40.047768 7203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:40.047930 7205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:40.048048 7202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:40.049170 7107 server_base.cc:1034] running on GCE node
I20260924 19:00:40.050107 7107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:40.051594 7107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:40.052798 7107 hybrid_clock.cc:648] HybridClock initialized: now 1790276440052755 us; error 45 us; skew 500 ppm
I20260924 19:00:40.053203 7107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:40 dist-test-slave-9fn1 krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276440, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:40.061432 7107 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260924 19:00:40.062942 7107 webserver.cc:460] Webserver started at http://127.0.97.60:33897/ using document root <none> and password file <none>
I20260924 19:00:40.063637 7107 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:40.063735 7107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:40.063977 7107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:40.066916 7107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-2/data/instance:
uuid: "3c7e501be5fb40ab92a39564889cadc9"
format_stamp: "Formatted at 2026-09-24 19:00:40 on dist-test-slave-9fn1"
server_key: "7cadc914a0cebc17d82e2aa7f344e9ac"
server_key_iv: "50d558f3c5e09e64373a5a62ac66789e"
server_key_version: "encryptionkey@0"
I20260924 19:00:40.067625 7107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-2/wal/instance:
uuid: "3c7e501be5fb40ab92a39564889cadc9"
format_stamp: "Formatted at 2026-09-24 19:00:40 on dist-test-slave-9fn1"
server_key: "7cadc914a0cebc17d82e2aa7f344e9ac"
server_key_iv: "50d558f3c5e09e64373a5a62ac66789e"
server_key_version: "encryptionkey@0"
I20260924 19:00:40.072602 7107 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 19:00:40.076404 7212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:40.077853 7107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:00:40.078032 7107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-2/wal
uuid: "3c7e501be5fb40ab92a39564889cadc9"
format_stamp: "Formatted at 2026-09-24 19:00:40 on dist-test-slave-9fn1"
server_key: "7cadc914a0cebc17d82e2aa7f344e9ac"
server_key_iv: "50d558f3c5e09e64373a5a62ac66789e"
server_key_version: "encryptionkey@0"
I20260924 19:00:40.078197 7107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:40.100540 7107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:40.104362 7107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:40.104630 7107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:40.126140 7265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:43355 every 8 connection(s)
I20260924 19:00:40.126173 7107 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:43355
I20260924 19:00:40.128065 7107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-2/data/info.pb
I20260924 19:00:40.130755 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.122707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45050 (local address 127.0.97.61:37383)
0924 19:00:40.123799 (+ 1092us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:40.123807 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:40.123829 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:00:40.123911 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:40.123917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:40.123992 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:40.124115 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:40.124543 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.125694 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.128344 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.129455 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.129622 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:40.129847 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:40.130265 (+ 418us) server_negotiation.cc:299] Negotiation successful
0924 19:00:40.130406 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":969,"thread_start_us":859,"threads_started":1}
I20260924 19:00:40.130772 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.122559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37383 (local address 127.0.0.1:45050)
0924 19:00:40.123592 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:40.123619 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:40.123765 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:40.124149 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:40.124154 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:40.124188 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:40.124396 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:40.124405 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.125892 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.125916 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:40.128145 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:40.128163 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.129708 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.129719 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:40.130029 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:40.130089 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 19:00:40.130237 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 19:00:40.130386 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"thread_start_us":726,"threads_started":1}
I20260924 19:00:40.132033 7158 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260924 19:00:40.132194 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 7107
I20260924 19:00:40.132601 7266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:40.132802 7186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f30647f1f9c43cf929f429a1e49e814. Latest consensus state: current_term: 1 leader_uuid: "8f30647f1f9c43cf929f429a1e49e814" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: true } } }
I20260924 19:00:40.133455 7186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [sys.catalog]: This master's current role is: LEARNER
I20260924 19:00:40.143187 7266 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:00:40 dist-test-slave-9fn1 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 24 19:00:40 dist-test-slave-9fn1 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276440, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 19:00:40.155480 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.134472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46208 (local address 127.0.97.60:43355)
0924 19:00:40.135341 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:40.135362 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:40.135412 (+ 50us) server_negotiation.cc:407] Connection header received
0924 19:00:40.137005 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:40.137033 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:40.137602 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:40.138094 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:40.139630 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.141638 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.142863 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.143564 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.143817 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:40.148744 (+ 4927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:40.148751 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:40.148763 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:40.148798 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:40.151610 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:40.152248 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:40.152264 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:40.152270 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:40.152414 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:40.152787 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:40.152794 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:40.152797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:40.153306 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:40.153625 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:40.154123 (+ 498us) server_negotiation.cc:299] Negotiation successful
0924 19:00:40.154526 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":411,"threads_started":1}
I20260924 19:00:40.161208 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.145972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33932 (local address 127.0.97.62:46537)
0924 19:00:40.146186 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:40.146193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:40.148408 (+ 2215us) server_negotiation.cc:407] Connection header received
0924 19:00:40.148577 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:40.148581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:40.148640 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:40.148758 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:40.150087 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.150893 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.152312 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.152709 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.152848 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:40.156198 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:40.156204 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:40.156208 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:40.156244 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:40.158967 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:40.159582 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:40.159586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:40.159588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:40.159708 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:40.159995 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:40.159998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:40.160000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:40.160255 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:40.160355 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:40.160752 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 19:00:40.160892 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 19:00:40.161641 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.145853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.0.1:33932)
0924 19:00:40.148266 (+ 2413us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:40.148317 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:40.148472 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:40.148808 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:40.148824 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:40.149661 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:40.149927 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:40.149936 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.151048 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.151053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:40.152173 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:40.152184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.152891 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.152897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:40.153074 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:40.153741 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:40.153763 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:40.156021 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:40.159096 (+ 3075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:40.159104 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:40.159108 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:40.159484 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:40.159796 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:40.159799 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:40.159801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:40.159911 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:40.160380 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:40.160386 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:40.160520 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 19:00:40.160683 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 19:00:40.160894 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2137,"mutex_wait_us":2,"thread_start_us":1987,"threads_started":1}
I20260924 19:00:40.162976 7266 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:00:40 dist-test-slave-9fn1 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276440, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260924 19:00:40.174582 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.163610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37383 (local address 127.0.0.1:45052)
0924 19:00:40.163788 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:40.163806 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:40.163898 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:40.164193 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:40.164196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:40.164464 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:40.164580 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:40.164587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.165466 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.165469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:40.166415 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:40.166425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.167116 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.167122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:40.167254 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:40.167732 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:40.167748 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:40.169860 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:40.172577 (+ 2717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:40.172580 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:40.172583 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:40.172926 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:40.173267 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:40.173270 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:40.173272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:40.173389 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:40.173887 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:40.173890 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:40.174032 (+ 142us) client_negotiation.cc:769] Sending connection context
0924 19:00:40.174137 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 19:00:40.174350 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 19:00:40.174614 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.163804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45052 (local address 127.0.97.61:37383)
0924 19:00:40.163968 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:40.163972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:40.163987 (+ 15us) server_negotiation.cc:407] Connection header received
0924 19:00:40.164031 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:40.164035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:40.164089 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:40.164163 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:40.164699 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.165345 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.166536 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.166940 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.167048 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:40.170048 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:40.170053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:40.170056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:40.170084 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:40.172465 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:40.173015 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:40.173019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:40.173021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:40.173152 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:40.173475 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:40.173479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:40.173480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:40.173793 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:40.173897 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:40.174231 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 19:00:40.174408 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 19:00:40.175652 7266 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43355 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:00:40 dist-test-slave-9fn1 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276440, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260924 19:00:40.187063 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.176228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43355 (local address 127.0.0.1:46224)
0924 19:00:40.176376 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:40.176403 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:40.176474 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:40.176700 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:40.176703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:40.177009 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:40.177154 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:40.177161 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.178080 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.178084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:40.179001 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:40.179011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.179656 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.179660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:40.179799 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:40.180317 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:40.180331 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:40.182696 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:40.185117 (+ 2421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:40.185120 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:40.185122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:40.185459 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:40.185826 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:40.185829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:40.185831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:40.185961 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:40.186371 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:40.186374 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:40.186474 (+ 100us) client_negotiation.cc:769] Sending connection context
0924 19:00:40.186603 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 19:00:40.186788 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 19:00:40.187224 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.176317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46224 (local address 127.0.97.60:43355)
0924 19:00:40.176447 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:40.176451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:40.176466 (+ 15us) server_negotiation.cc:407] Connection header received
0924 19:00:40.176542 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:40.176545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:40.176592 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:40.176664 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:40.177268 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.177979 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.179105 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.179505 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.179636 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:40.182851 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:40.182856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:40.182859 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:40.182885 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:40.185011 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:40.185613 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:40.185617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:40.185619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:40.185736 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:40.186044 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:40.186047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:40.186048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:40.186287 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:40.186382 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:40.186694 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 19:00:40.186914 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":42}
I20260924 19:00:40.191651 7266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9: Bootstrap starting.
I20260924 19:00:40.194559 7266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:40.195497 7266 log.cc:824] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:40.197898 7266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9: No bootstrap required, opened a new log
I20260924 19:00:40.204617 7266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43355 } }
I20260924 19:00:40.204871 7266 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:40.204939 7266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c7e501be5fb40ab92a39564889cadc9, State: Initialized, Role: FOLLOWER
I20260924 19:00:40.205536 7266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43355 } }
I20260924 19:00:40.206358 7272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43355 } } }
I20260924 19:00:40.206636 7272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:00:40.207319 7266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 19:00:40.216872 7283 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:00:40.217017 7283 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:00:40.220816 7107 master_runner.cc:428] Detected that this master 3c7e501be5fb40ab92a39564889cadc9 is joining an existing cluster
I20260924 19:00:40.220908 7107 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:43355
I20260924 19:00:40.241192 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.225795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33944 (local address 127.0.97.62:46537)
0924 19:00:40.226052 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:40.226059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:40.226460 (+ 401us) server_negotiation.cc:407] Connection header received
0924 19:00:40.226662 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:40.226668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:40.226753 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:40.226856 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:40.227878 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.228588 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.230169 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.230851 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.230941 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:40.234389 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:40.234396 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:40.234401 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:40.234436 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:40.237920 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:40.238769 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:40.238774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:40.238778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:40.238920 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:40.239408 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:40.239414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:40.239417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:40.239759 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:40.239891 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:40.240605 (+ 714us) server_negotiation.cc:299] Negotiation successful
0924 19:00:40.240843 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260924 19:00:40.243702 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.225276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.0.1:33944)
0924 19:00:40.226362 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:40.226377 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:40.226529 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:40.226880 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:40.226885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:40.227463 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:40.227741 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:40.227751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.228711 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.228718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:40.229963 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:40.229981 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.231240 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.231252 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:40.231446 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:40.232995 (+ 1549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:40.233023 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:40.234169 (+ 1146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:40.238122 (+ 3953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:40.238127 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:40.238131 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:40.238616 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:40.239064 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:40.239068 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:40.239072 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:40.239270 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:40.239970 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:40.239975 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:40.240247 (+ 272us) client_negotiation.cc:769] Sending connection context
0924 19:00:40.242436 (+ 2189us) client_negotiation.cc:240] Negotiation successful
0924 19:00:40.242787 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":985,"thread_start_us":4634,"threads_started":1}
I20260924 19:00:40.249918 7289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.225279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37383 (local address 127.0.0.1:45064)
0924 19:00:40.230276 (+ 4997us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:40.230294 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:40.230449 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:40.230942 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:40.230947 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:40.231606 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:40.232700 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:40.232717 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.234648 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.234655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:40.236910 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:40.236934 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.238508 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.238517 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:40.238754 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:40.239551 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:40.239581 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:40.241066 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:40.245970 (+ 4904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:40.245982 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:40.245988 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:40.246645 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:40.247193 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:40.247201 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:40.247207 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:40.247439 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:40.248282 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:40.248290 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:40.248581 (+ 291us) client_negotiation.cc:769] Sending connection context
0924 19:00:40.248763 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 19:00:40.249216 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4876,"mutex_wait_us":70,"thread_start_us":4176,"threads_started":1}
I20260924 19:00:40.251139 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.225513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45064 (local address 127.0.97.61:37383)
0924 19:00:40.225735 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:40.225741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:40.230360 (+ 4619us) server_negotiation.cc:407] Connection header received
0924 19:00:40.230590 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:40.230597 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:40.230671 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:40.230749 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:40.232942 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.234426 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.237169 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.238013 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.238174 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:40.241345 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:40.241355 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:40.241362 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:40.241403 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:40.245716 (+ 4313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:40.246833 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:40.246842 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:40.246848 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:40.247027 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:40.247593 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:40.247601 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:40.247607 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:40.248031 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:40.248195 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:40.250335 (+ 2140us) server_negotiation.cc:299] Negotiation successful
0924 19:00:40.250645 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 19:00:40.258926 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.230528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46232 (local address 127.0.97.60:43355)
0924 19:00:40.230705 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:40.230711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:40.244023 (+ 13312us) server_negotiation.cc:407] Connection header received
0924 19:00:40.244240 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:40.244246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:40.244319 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:40.244385 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:40.245409 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.246567 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.248430 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.249123 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.249259 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:40.251656 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:40.251663 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:40.251681 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:40.251715 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:40.255903 (+ 4188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:40.256702 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:40.256708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:40.256712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:40.256856 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:40.257292 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:40.257298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:40.257306 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:40.257675 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:40.257811 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:40.258360 (+ 549us) server_negotiation.cc:299] Negotiation successful
0924 19:00:40.258557 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 19:00:40.260311 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.230804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43355 (local address 127.0.0.1:46232)
0924 19:00:40.243938 (+ 13134us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:40.243958 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:40.244103 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:40.244559 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:40.244564 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:40.245046 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:40.245258 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:40.245269 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.246787 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.246792 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:40.248266 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:40.248280 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.249539 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.249545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:40.249703 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:40.250341 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:40.250368 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:40.251454 (+ 1086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:40.256074 (+ 4620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:40.256079 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:40.256083 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:40.256582 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:40.256986 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:40.256990 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:40.256995 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:40.257153 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:40.257911 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:40.257915 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:40.258050 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 19:00:40.259622 (+ 1572us) client_negotiation.cc:240] Negotiation successful
0924 19:00:40.259882 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13032,"thread_start_us":22129,"threads_started":1}
I20260924 19:00:40.261222 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 6611
Sep 24 19:00:40 dist-test-slave-9fn1 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276437, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260924 19:00:40.278746 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.258089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43355 (local address 127.0.0.1:46240)
0924 19:00:40.258437 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:40.258461 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:40.258589 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:40.259553 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:40.259561 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:40.260097 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:40.260326 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:40.260338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.262613 (+ 2275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.262622 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:40.264489 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:40.264509 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.265766 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.265771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:40.265942 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:40.266629 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:40.266650 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:40.270466 (+ 3816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:40.276210 (+ 5744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:40.276219 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:40.276223 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:40.276613 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:40.276974 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:40.276979 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:40.276983 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:40.277126 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:40.277756 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:40.277764 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:40.277933 (+ 169us) client_negotiation.cc:769] Sending connection context
0924 19:00:40.278074 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 19:00:40.278347 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197}
I20260924 19:00:40.279672 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.258107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46240 (local address 127.0.97.60:43355)
0924 19:00:40.259153 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:40.259158 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:40.259174 (+ 16us) server_negotiation.cc:407] Connection header received
0924 19:00:40.259274 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:40.259279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:40.259381 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:40.259481 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:40.260722 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.262412 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.264728 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.265576 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.270690 (+ 5114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:40.270961 (+ 271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:40.270969 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:40.270977 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:40.271016 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:40.276023 (+ 5007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:40.276746 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:40.276749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:40.276751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:40.276848 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:40.277243 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:40.277250 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:40.277254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:40.277598 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:40.279122 (+ 1524us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:40.279234 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 19:00:40.279419 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":952,"thread_start_us":18025,"threads_started":1}
I20260924 19:00:40.281322 6550 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:43355: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 43355
}
attrs {
promote: true
}
}
dest_uuid: "8f30647f1f9c43cf929f429a1e49e814"
cas_config_opid_index: 5
I20260924 19:00:40.282377 6550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: true } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: true } }
I20260924 19:00:40.284049 7110 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3c7e501be5fb40ab92a39564889cadc9 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: true } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: true } } }
I20260924 19:00:40.288863 7110 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:43355
Sep 24 19:00:40 dist-test-slave-9fn1 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260924 19:00:40.290038 7107 master.cc:537] Master@127.0.97.60:43355 shutting down...
I20260924 19:00:40.290948 7158 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [term 1 LEARNER]: Refusing update from remote peer 8f30647f1f9c43cf929f429a1e49e814: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 19:00:40.297179 7295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8f30647f1f9c43cf929f429a1e49e814" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: true } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: true } } }
I20260924 19:00:40.297871 7295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:40.296957 7294 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20260924 19:00:40.300971 7293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.262985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40206 (local address 127.0.97.1:33575)
0924 19:00:40.278017 (+ 15032us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:40.278028 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:40.278061 (+ 33us) server_negotiation.cc:407] Connection header received
0924 19:00:40.278136 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:40.278144 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:40.278429 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:40.278605 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:40.280146 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.281646 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.283870 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.284723 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.292123 (+ 7400us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:40.292398 (+ 275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:40.292409 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:40.292426 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:40.292463 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:40.296207 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:40.297837 (+ 1630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:40.297845 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:40.297848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:40.297996 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:40.298450 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:40.298455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:40.298458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:40.298853 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:40.300179 (+ 1326us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:40.300344 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 19:00:40.300584 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14857,"thread_start_us":14703,"threads_started":1}
I20260924 19:00:40.302282 6734 generic_service.cc:196] Checking for leaks (request via RPC)
W20260924 19:00:40.305958 6535 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 -> Peer 3c7e501be5fb40ab92a39564889cadc9 (127.0.97.60:43355): Couldn't send request to peer 3c7e501be5fb40ab92a39564889cadc9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:00:40.311023 7158 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 3c7e501be5fb40ab92a39564889cadc9 (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: true } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: true } } }
I20260924 19:00:40.316602 7186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f30647f1f9c43cf929f429a1e49e814" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: true } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: true } } }
I20260924 19:00:40.316888 7186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [sys.catalog]: This master's current role is: LEARNER
I20260924 19:00:40.320938 7294 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814: attempting to promote NON_VOTER 07bd526e5ab649bb873f289aee41b508 to VOTER
I20260924 19:00:40.321820 7107 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:00:40.322220 7107 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:40.322427 7107 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9: stopping tablet replica
I20260924 19:00:40.322656 7294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: true } }
I20260924 19:00:40.329522 7107 master.cc:552] Master@127.0.97.60:43355 shutdown complete.
I20260924 19:00:40.329651 7107 master_runner.cc:305] Clearing existing system tablet
I20260924 19:00:40.330070 7107 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:40.330222 7107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:40.331384 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.330838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43355 (local address 127.0.0.1:46254)
0924 19:00:40.331069 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:40.331212 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260924 19:00:40.332508 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.332049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43355 (local address 127.0.0.1:46262)
0924 19:00:40.332298 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:40.332380 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260924 19:00:40.332696 6535 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 -> Peer 3c7e501be5fb40ab92a39564889cadc9 (127.0.97.60:43355): Couldn't send request to peer 3c7e501be5fb40ab92a39564889cadc9. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.60:43355: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 19:00:40.333164 7158 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [term 1 LEARNER]: Refusing update from remote peer 8f30647f1f9c43cf929f429a1e49e814: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 19:00:40.333730 7300 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 19:00:40.335953 7107 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20260924 19:00:40.336256 7294 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 07bd526e5ab649bb873f289aee41b508 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: true } } }
I20260924 19:00:40.340353 7295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8f30647f1f9c43cf929f429a1e49e814" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: true } } }
I20260924 19:00:40.340576 7295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:40.341421 7158 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 07bd526e5ab649bb873f289aee41b508 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: true } } }
I20260924 19:00:40.345265 7186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f30647f1f9c43cf929f429a1e49e814" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: true } } }
I20260924 19:00:40.345489 7186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:00:40.346161 7295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f30647f1f9c43cf929f429a1e49e814" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: true } } }
I20260924 19:00:40.346379 7295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:40.349932 7303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:40.350281 7107 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:40.350553 7107 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.005s sys 0.007s
I20260924 19:00:40.350657 7107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-2/wal
uuid: "3c7e501be5fb40ab92a39564889cadc9"
format_stamp: "Formatted at 2026-09-24 19:00:40 on dist-test-slave-9fn1"
server_key: "7cadc914a0cebc17d82e2aa7f344e9ac"
server_key_iv: "50d558f3c5e09e64373a5a62ac66789e"
server_key_version: "encryptionkey@0"
I20260924 19:00:40.355552 7107 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:40.362478 7107 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 19:00:40.362625 7107 log.cc:1196] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 19:00:40.363051 7107 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9: Deleting consensus metadata
I20260924 19:00:40.363296 7107 master_runner.cc:315] Copying system tablet from 127.0.97.62:46537
I20260924 19:00:40.365669 7107 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:46537
I20260924 19:00:40.385083 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.366493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33956 (local address 127.0.97.62:46537)
0924 19:00:40.366705 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:40.366712 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:40.370123 (+ 3411us) server_negotiation.cc:407] Connection header received
0924 19:00:40.370252 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:40.370259 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:40.370338 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:40.370446 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:40.371462 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.372659 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.374628 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.375340 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.375444 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:40.378355 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:40.378362 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:40.378366 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:40.378414 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:40.381936 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:40.382754 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:40.382760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:40.382764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:40.382908 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:40.383401 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:40.383407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:40.383411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:40.383757 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:40.383885 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:40.384471 (+ 586us) server_negotiation.cc:299] Negotiation successful
0924 19:00:40.384683 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260924 19:00:40.386333 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.366475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.0.1:33956)
0924 19:00:40.369935 (+ 3460us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:40.369955 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:40.370113 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:40.370490 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:40.370495 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:40.371037 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:40.371310 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:40.371322 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.372835 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.372841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:40.374454 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:40.374469 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.375702 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.375708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:40.376286 (+ 578us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:40.377010 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:40.377031 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:40.378124 (+ 1093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:40.382158 (+ 4034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:40.382163 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:40.382167 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:40.382622 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:40.383108 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:40.383113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:40.383118 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:40.383283 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:40.383960 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:40.383965 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:40.384183 (+ 218us) client_negotiation.cc:769] Sending connection context
0924 19:00:40.385527 (+ 1344us) client_negotiation.cc:240] Negotiation successful
0924 19:00:40.385842 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3289,"thread_start_us":3163,"threads_started":1}
I20260924 19:00:40.387073 6570 tablet_copy_service.cc:140] P 8f30647f1f9c43cf929f429a1e49e814: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3c7e501be5fb40ab92a39564889cadc9 ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:33956)
I20260924 19:00:40.387248 6570 tablet_copy_service.cc:161] P 8f30647f1f9c43cf929f429a1e49e814: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3c7e501be5fb40ab92a39564889cadc9 at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:33956: session id = 3c7e501be5fb40ab92a39564889cadc9-00000000000000000000000000000000
I20260924 19:00:40.391268 6570 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 19:00:40.393013 7107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 19:00:40.399488 7107 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9: tablet copy: Starting download of 0 data blocks...
I20260924 19:00:40.399762 7107 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9: tablet copy: Starting download of 1 WAL segments...
I20260924 19:00:40.402105 7107 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 19:00:40.406483 6570 tablet_copy_service.cc:342] P 8f30647f1f9c43cf929f429a1e49e814: Request end of tablet copy session 3c7e501be5fb40ab92a39564889cadc9-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:33956
I20260924 19:00:40.406613 6570 tablet_copy_service.cc:433] P 8f30647f1f9c43cf929f429a1e49e814: ending tablet copy session 3c7e501be5fb40ab92a39564889cadc9-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3c7e501be5fb40ab92a39564889cadc9
I20260924 19:00:40.417371 7107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:40.420050 7310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:40.420429 7311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:40.425343 7107 server_base.cc:1034] running on GCE node
W20260924 19:00:40.425450 7313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:40.425935 7107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:40.426891 7107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:40.428102 7107 hybrid_clock.cc:648] HybridClock initialized: now 1790276440428058 us; error 34 us; skew 500 ppm
I20260924 19:00:40.428335 7107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:00:40.429360 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 6611
Sep 24 19:00:40 dist-test-slave-9fn1 krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276440, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:40.431128 7107 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260924 19:00:40.437996 7107 webserver.cc:460] Webserver started at http://127.0.97.60:44451/ using document root <none> and password file <none>
I20260924 19:00:40.438416 7107 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:40.438519 7107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:40.441394 7107 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 19:00:40.443235 7320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:40.443689 7107 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 19:00:40.443810 7107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-2/wal
uuid: "3c7e501be5fb40ab92a39564889cadc9"
format_stamp: "Formatted at 2026-09-24 19:00:40 on dist-test-slave-9fn1"
server_key: "7cadc914a0cebc17d82e2aa7f344e9ac"
server_key_iv: "50d558f3c5e09e64373a5a62ac66789e"
server_key_version: "encryptionkey@0"
I20260924 19:00:40.443979 7107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:40.449172 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:33575
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46537
--webserver_port=37121
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:46537,127.0.97.61:37383
--tserver_master_addrs=127.0.97.62:46537,127.0.97.61:37383,127.0.97.60:43355 with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:00:40.461074 7107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:40.465359 7107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:40.499997 7107 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:43355
I20260924 19:00:40.500300 7372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:43355 every 8 connection(s)
I20260924 19:00:40.501242 7107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/master-2/data/info.pb
I20260924 19:00:40.507139 7373 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:00:40.512076 7373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9: Bootstrap starting.
I20260924 19:00:40.530288 7373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:00:40.530921 7373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9: Bootstrap complete.
I20260924 19:00:40.532094 7373 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: true } }
I20260924 19:00:40.532506 7373 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3c7e501be5fb40ab92a39564889cadc9, State: Initialized, Role: LEARNER
I20260924 19:00:40.532912 7373 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: true } }
I20260924 19:00:40.533825 7376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: true } } }
I20260924 19:00:40.534113 7373 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:00:40.534169 7376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [sys.catalog]: This master's current role is: LEARNER
I20260924 19:00:40.543159 7387 catalog_manager.cc:1259] Loaded cluster ID: d0f1f7a463e442e28820eee13444c245
I20260924 19:00:40.543257 7387 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9: loading cluster ID for follower catalog manager: success
I20260924 19:00:40.546591 7387 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9: acquiring CA information for follower catalog manager: success
I20260924 19:00:40.548722 7387 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 19:00:40.806563 7325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:40.806913 7325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:40.807101 7325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:40.817858 7325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:40.818005 7325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:40.818187 7325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:00:40.828424 7325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:33575
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=37121
--tserver_master_addrs=127.0.97.62:46537,127.0.97.61:37383,127.0.97.60:43355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:40.829823 7325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:40.831149 7325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:40.848327 7393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:40.849064 7395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:40.849592 7392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:40.855079 7325 server_base.cc:1034] running on GCE node
I20260924 19:00:40.856015 7325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:40.857364 7325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:40.858624 7325 hybrid_clock.cc:648] HybridClock initialized: now 1790276440858537 us; error 91 us; skew 500 ppm
I20260924 19:00:40.859086 7325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:40 dist-test-slave-9fn1 krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276440, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:40.863168 7325 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260924 19:00:40.864913 7325 webserver.cc:460] Webserver started at http://127.0.97.1:37121/ using document root <none> and password file <none>
I20260924 19:00:40.865962 7325 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:40.866119 7325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:40.879611 7325 fs_manager.cc:709] Time spent opening directory manager: real 0.010s user 0.004s sys 0.004s
I20260924 19:00:40.882874 7404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:40.884549 7325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:00:40.884737 7325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "31b40f242c874e81a6ab82ce8cb210c7"
format_stamp: "Formatted at 2026-09-24 19:00:37 on dist-test-slave-9fn1"
server_key: "f76bb72716da4356a5119834c4d72a18"
server_key_iv: "3e45165b206e313a38dda5a03539ec9f"
server_key_version: "encryptionkey@0"
I20260924 19:00:40.885829 7325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:40.887135 7402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.865056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46270 (local address 127.0.97.60:43355)
0924 19:00:40.877541 (+ 12485us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:40.877551 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:40.878455 (+ 904us) server_negotiation.cc:407] Connection header received
0924 19:00:40.878698 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:40.878706 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:40.878845 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:40.878988 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:40.879497 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.881764 (+ 2267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.883921 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.885560 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.885709 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:40.885962 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:40.886599 (+ 637us) server_negotiation.cc:299] Negotiation successful
0924 19:00:40.886753 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12258,"thread_start_us":12495,"threads_started":1}
I20260924 19:00:40.887149 7401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.864771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43355 (local address 127.0.0.1:46270)
0924 19:00:40.878298 (+ 13527us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:40.878326 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:40.878506 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:40.879060 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:40.879065 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:40.879092 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:40.879288 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:40.879298 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.881951 (+ 2653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.881957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:40.883704 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:40.883721 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:40.886086 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:40.886095 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:40.886353 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:40.886408 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 19:00:40.886527 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 19:00:40.886713 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13411,"thread_start_us":13327,"threads_started":1}
I20260924 19:00:40.888304 7348 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 19:00:40.888962 7376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f30647f1f9c43cf929f429a1e49e814. Latest consensus state: current_term: 1 leader_uuid: "8f30647f1f9c43cf929f429a1e49e814" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: true } } }
I20260924 19:00:40.889305 7376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [sys.catalog]: This master's current role is: LEARNER
I20260924 19:00:40.907388 7325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:40.911613 7325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:40.912385 7325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:40.913403 7325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:40.914956 7325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:40.915087 7325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:40.915216 7325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:40.915300 7325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:40.963749 7325 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33575
I20260924 19:00:40.963806 7516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33575 every 8 connection(s)
I20260924 19:00:40.965263 7325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:00:40.971932 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 7325
I20260924 19:00:40.972368 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 6745
Sep 24 19:00:41 dist-test-slave-9fn1 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276440, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 24 19:00:41 dist-test-slave-9fn1 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276440, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 24 19:00:41 dist-test-slave-9fn1 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260924 19:00:41.018860 7402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.971019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54167 (local address 127.0.97.60:43355)
0924 19:00:40.973079 (+ 2060us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:40.973090 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:40.977164 (+ 4074us) server_negotiation.cc:407] Connection header received
0924 19:00:40.996210 (+ 19046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:40.996219 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:40.996319 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:40.996428 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:41.000097 (+ 3669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:41.001246 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:41.004210 (+ 2964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:41.004938 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:41.005060 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:41.010610 (+ 5550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:41.010617 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:41.010623 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:41.010675 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:41.014720 (+ 4045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:41.015794 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:41.015800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:41.015804 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:41.015957 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:41.016556 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:41.016562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:41.016565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:41.016934 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:41.017102 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:41.018233 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0924 19:00:41.018485 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1860}
I20260924 19:00:41.021579 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.969074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43355 (local address 127.0.97.1:54167)
0924 19:00:40.976884 (+ 7810us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:40.976960 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:40.995796 (+ 18836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:40.996882 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:40.996897 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:40.998293 (+ 1396us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:40.999871 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:40.999898 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:41.001451 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:41.001457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:41.004030 (+ 2573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:41.004045 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:41.005404 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:41.005413 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:41.005786 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:41.006636 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:41.006677 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:41.010341 (+ 3664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:41.014888 (+ 4547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:41.014897 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:41.014916 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:41.015633 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:41.016134 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:41.016139 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:41.016143 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:41.016411 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:41.017215 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:41.017225 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:41.017712 (+ 487us) client_negotiation.cc:769] Sending connection context
0924 19:00:41.019467 (+ 1755us) client_negotiation.cc:240] Negotiation successful
0924 19:00:41.020049 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7264,"mutex_wait_us":224,"spinlock_wait_cycles":1280,"thread_start_us":7981,"threads_started":1}
I20260924 19:00:41.021909 7526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.978647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56343 (local address 127.0.97.62:46537)
0924 19:00:40.994998 (+ 16351us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:40.995004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:40.995022 (+ 18us) server_negotiation.cc:407] Connection header received
0924 19:00:40.996193 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:40.996198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:40.996276 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:40.996389 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:41.000085 (+ 3696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:41.001403 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:41.004237 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:41.005046 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:41.005159 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:41.010884 (+ 5725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:41.010892 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:41.010900 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:41.010947 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:41.016373 (+ 5426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:41.019289 (+ 2916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:41.019297 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:41.019302 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:41.019463 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:41.019955 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:41.019963 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:41.019969 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:41.020332 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:41.020481 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:41.021070 (+ 589us) server_negotiation.cc:299] Negotiation successful
0924 19:00:41.021330 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16227,"spinlock_wait_cycles":10496,"thread_start_us":16205,"threads_started":1}
I20260924 19:00:41.023078 7527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.983199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.97.1:56343)
0924 19:00:40.993036 (+ 9837us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:40.993059 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:40.995789 (+ 2730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:40.996881 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:40.996896 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:40.998292 (+ 1396us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:40.999872 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:40.999898 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:41.001626 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:41.001634 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:41.004062 (+ 2428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:41.004079 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:41.005485 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:41.005493 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:41.005789 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:41.007027 (+ 1238us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:41.007054 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:41.010662 (+ 3608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:41.016542 (+ 5880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:41.016549 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:41.016555 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:41.017176 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:41.019632 (+ 2456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:41.019638 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:41.019644 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:41.019824 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:41.020579 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:41.020592 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:41.020764 (+ 172us) client_negotiation.cc:769] Sending connection context
0924 19:00:41.022179 (+ 1415us) client_negotiation.cc:240] Negotiation successful
0924 19:00:41.022497 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9684,"mutex_wait_us":874,"spinlock_wait_cycles":29952,"thread_start_us":9989,"threads_started":1}
I20260924 19:00:41.023859 7517 heartbeater.cc:347] Connected to a master server at 127.0.97.60:43355
I20260924 19:00:41.024185 7517 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:41.025944 7517 heartbeater.cc:511] Master 127.0.97.60:43355 requested a full tablet report, sending...
I20260924 19:00:41.026036 7518 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46537
I20260924 19:00:41.026181 7518 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:41.026640 7518 heartbeater.cc:511] Master 127.0.97.62:46537 requested a full tablet report, sending...
I20260924 19:00:41.027166 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.974173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34610 (local address 127.0.97.2:44407)
0924 19:00:40.995463 (+ 21290us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:40.995472 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:40.995501 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:00:40.995572 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:40.995580 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:40.995860 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:40.996029 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:40.998364 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:41.000492 (+ 2128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:41.002482 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:41.003373 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:41.003499 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:41.012477 (+ 8978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:41.012495 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:41.012529 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:41.012878 (+ 349us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:41.021459 (+ 8581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:41.022583 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:41.022594 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:41.022598 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:41.022773 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:41.023615 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:41.023624 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:41.023631 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:41.024145 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:41.024490 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:41.025294 (+ 804us) server_negotiation.cc:299] Negotiation successful
0924 19:00:41.025754 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21116,"thread_start_us":23932,"threads_started":1}
I20260924 19:00:41.028012 6550 ts_manager.cc:194] Re-registered known tserver with Master: 31b40f242c874e81a6ab82ce8cb210c7 (127.0.97.1:33575)
I20260924 19:00:41.028445 6868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:41.029315 6550 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56343
I20260924 19:00:41.029594 7338 ts_manager.cc:194] Registered new tserver with Master: 31b40f242c874e81a6ab82ce8cb210c7 (127.0.97.1:33575)
Sep 24 19:00:41 dist-test-slave-9fn1 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276440, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260924 19:00:41.042825 7523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.970596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49863 (local address 127.0.97.61:37383)
0924 19:00:40.992537 (+ 21941us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:40.992543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:40.992571 (+ 28us) server_negotiation.cc:407] Connection header received
0924 19:00:41.005933 (+ 13362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:41.005943 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:41.006068 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:41.006152 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:41.011876 (+ 5724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:41.013301 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:41.023943 (+ 10642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:41.024688 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:41.024799 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:41.036420 (+ 11621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:41.036426 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:41.036430 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:41.036466 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:41.040099 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:41.040754 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:41.040760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:41.040762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:41.040882 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:41.041297 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:41.041307 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:41.041309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:41.041638 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:41.041828 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:41.042182 (+ 354us) server_negotiation.cc:299] Negotiation successful
0924 19:00:41.042399 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21779,"thread_start_us":21644,"threads_started":1}
I20260924 19:00:41.042851 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:40.969074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37383 (local address 127.0.97.1:49863)
0924 19:00:40.980393 (+ 11319us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:40.980418 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:40.999387 (+ 18969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:41.006779 (+ 7392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:41.006786 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:41.009410 (+ 2624us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:41.009766 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:41.009780 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:41.014549 (+ 4769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:41.014557 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:41.019103 (+ 4546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:41.019120 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:41.031754 (+ 12634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:41.031769 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:41.032121 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:41.032927 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:41.032962 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:41.036156 (+ 3194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:41.040243 (+ 4087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:41.040251 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:41.040256 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:41.040608 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:41.041013 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:41.041018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:41.041022 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:41.041171 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:41.041770 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:41.041778 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:41.041931 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 19:00:41.042080 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 19:00:41.042412 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11114,"mutex_wait_us":224,"thread_start_us":10686,"threads_started":1}
I20260924 19:00:41.044612 7519 heartbeater.cc:347] Connected to a master server at 127.0.97.61:37383
I20260924 19:00:41.044786 7519 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:41.045270 7519 heartbeater.cc:511] Master 127.0.97.61:37383 requested a full tablet report, sending...
I20260924 19:00:41.046945 7148 ts_manager.cc:194] Registered new tserver with Master: 31b40f242c874e81a6ab82ce8cb210c7 (127.0.97.1:33575)
I20260924 19:00:41.152433 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 6745
I20260924 19:00:41.168397 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:44407
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46537
--webserver_port=44301
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:46537,127.0.97.61:37383
--tserver_master_addrs=127.0.97.62:46537,127.0.97.61:37383,127.0.97.60:43355 with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:00:41.409956 7300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f30647f1f9c43cf929f429a1e49e814" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: true } } }
I20260924 19:00:41.410362 7300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:41.411672 7531 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814: attempting to promote NON_VOTER 3c7e501be5fb40ab92a39564889cadc9 to VOTER
I20260924 19:00:41.412407 7531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: false } }
I20260924 19:00:41.414724 7158 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [term 1 FOLLOWER]: Refusing update from remote peer 8f30647f1f9c43cf929f429a1e49e814: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 19:00:41.414855 7348 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [term 1 LEARNER]: Refusing update from remote peer 8f30647f1f9c43cf929f429a1e49e814: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 19:00:41.415307 7530 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 19:00:41.415791 7531 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 19:00:41.418634 7530 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3c7e501be5fb40ab92a39564889cadc9 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: false } } }
I20260924 19:00:41.420392 7348 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3c7e501be5fb40ab92a39564889cadc9 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: false } } }
I20260924 19:00:41.422312 7158 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3c7e501be5fb40ab92a39564889cadc9 (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: false } } }
I20260924 19:00:41.422477 7530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "8f30647f1f9c43cf929f429a1e49e814" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: false } } }
I20260924 19:00:41.422809 7530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:41.423059 7531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f30647f1f9c43cf929f429a1e49e814" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: false } } }
I20260924 19:00:41.423309 7531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:41.423596 7531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f30647f1f9c43cf929f429a1e49e814" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: false } } }
I20260924 19:00:41.423840 7531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814 [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:41.429654 7536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f30647f1f9c43cf929f429a1e49e814" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: false } } }
I20260924 19:00:41.430007 7536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:00:41.430598 7537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f30647f1f9c43cf929f429a1e49e814" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: false } } }
I20260924 19:00:41.430982 7537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 19:00:41.501288 7529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:41.501631 7529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:41.501780 7529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:41.508810 7529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:41.508909 7529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:41.509094 7529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:00:41.519073 7529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:44407
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=44301
--tserver_master_addrs=127.0.97.62:46537,127.0.97.61:37383,127.0.97.60:43355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:41.520418 7529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:41.521795 7529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:41.535511 7543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:41.535437 7542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:41.535554 7545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:41.541375 7529 server_base.cc:1034] running on GCE node
I20260924 19:00:41.542088 7529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:41.543009 7529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:41.544281 7529 hybrid_clock.cc:648] HybridClock initialized: now 1790276441544175 us; error 71 us; skew 500 ppm
I20260924 19:00:41.544590 7529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:41 dist-test-slave-9fn1 krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276441, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:41.547809 7529 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260924 19:00:41.549326 7529 webserver.cc:460] Webserver started at http://127.0.97.2:44301/ using document root <none> and password file <none>
I20260924 19:00:41.550114 7529 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:41.550278 7529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:41.555447 7529 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 19:00:41.558120 7552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:41.559393 7529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:00:41.559568 7529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "56fd3560adc24fa5aa71678e700f9836"
format_stamp: "Formatted at 2026-09-24 19:00:38 on dist-test-slave-9fn1"
server_key: "1d5ce75eec9c72a6cd432501ddd9f6b6"
server_key_iv: "29af93175fa4ae99d21bee60b4eccacf"
server_key_version: "encryptionkey@0"
I20260924 19:00:41.560498 7529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:41.575340 7529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:41.579826 7529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:41.580099 7529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:41.580992 7529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:41.582484 7529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:41.582634 7529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:41.582758 7529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:41.582849 7529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:41.615744 7529 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44407
I20260924 19:00:41.615797 7664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44407 every 8 connection(s)
I20260924 19:00:41.617285 7529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:00:41.618461 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 7529
I20260924 19:00:41.618764 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 6878
Sep 24 19:00:41 dist-test-slave-9fn1 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276441, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 24 19:00:41 dist-test-slave-9fn1 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276441, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 24 19:00:41 dist-test-slave-9fn1 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276441, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 24 19:00:41 dist-test-slave-9fn1 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260924 19:00:41.675546 7673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:41.621554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52835 (local address 127.0.97.60:43355)
0924 19:00:41.644213 (+ 22659us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:41.644239 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:41.650784 (+ 6545us) server_negotiation.cc:407] Connection header received
0924 19:00:41.650868 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:41.650876 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:41.650992 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:41.651112 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:41.656253 (+ 5141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:41.657362 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:41.660370 (+ 3008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:41.660973 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:41.661069 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:41.667306 (+ 6237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:41.667315 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:41.667321 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:41.667367 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:41.671500 (+ 4133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:41.672980 (+ 1480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:41.672987 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:41.672991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:41.673135 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:41.673579 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:41.673586 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:41.673591 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:41.673964 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:41.674106 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:41.674876 (+ 770us) server_negotiation.cc:299] Negotiation successful
0924 19:00:41.675125 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22470,"thread_start_us":22703,"threads_started":1}
I20260924 19:00:41.676837 7672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:41.621345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56147 (local address 127.0.97.61:37383)
0924 19:00:41.649791 (+ 28446us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:41.649799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:41.649827 (+ 28us) server_negotiation.cc:407] Connection header received
0924 19:00:41.656860 (+ 7033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:41.656867 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:41.656975 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:41.657082 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:41.658544 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:41.659900 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:41.661953 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:41.662633 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:41.662751 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:41.667638 (+ 4887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:41.667646 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:41.667650 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:41.667716 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:41.670889 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:41.671700 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:41.671706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:41.671709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:41.671858 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:41.672873 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:41.672880 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:41.672885 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:41.673247 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:41.673399 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:41.676060 (+ 2661us) server_negotiation.cc:299] Negotiation successful
0924 19:00:41.676458 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28304,"thread_start_us":28221,"threads_started":1}
I20260924 19:00:41.678247 7674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:41.620828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37383 (local address 127.0.97.2:56147)
0924 19:00:41.646134 (+ 25306us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:41.646192 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:41.649394 (+ 3202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:41.657165 (+ 7771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:41.657170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:41.657861 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:41.658305 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:41.658321 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:41.660077 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:41.660082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:41.661797 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:41.661811 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:41.663008 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:41.663014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:41.663233 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:41.663920 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:41.663943 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:41.667396 (+ 3453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:41.671021 (+ 3625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:41.671030 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:41.671049 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:41.671552 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:41.672002 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:41.672007 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:41.672011 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:41.672726 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:41.673485 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:41.673494 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:41.674008 (+ 514us) client_negotiation.cc:769] Sending connection context
0924 19:00:41.674343 (+ 335us) client_negotiation.cc:240] Negotiation successful
0924 19:00:41.674893 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24981,"mutex_wait_us":27,"thread_start_us":28972,"threads_started":1}
I20260924 19:00:41.678282 7670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:41.620828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43355 (local address 127.0.97.2:52835)
0924 19:00:41.650251 (+ 29423us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:41.650274 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:41.650425 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:41.651666 (+ 1241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:41.651683 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:41.652917 (+ 1234us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:41.655983 (+ 3066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:41.656009 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:41.657566 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:41.657571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:41.660202 (+ 2631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:41.660219 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:41.661446 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:41.661453 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:41.661786 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:41.662577 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:41.662610 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:41.667023 (+ 4413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:41.671640 (+ 4617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:41.671645 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:41.671648 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:41.672793 (+ 1145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:41.673271 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:41.673276 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:41.673279 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:41.673405 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:41.674170 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:41.674175 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:41.674322 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 19:00:41.674470 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 19:00:41.677190 (+ 2720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29244,"mutex_wait_us":2940,"thread_start_us":24163,"threads_started":1}
I20260924 19:00:41.679507 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:41.646876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.97.2:41545)
0924 19:00:41.657264 (+ 10388us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:41.657281 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:41.657380 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:41.660450 (+ 3070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:41.660454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:41.661056 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:41.661413 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:41.661424 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:41.663271 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:41.663276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:41.665570 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:41.665584 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:41.666603 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:41.666609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:41.666808 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:41.667477 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:41.667499 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:41.672793 (+ 5294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:41.676332 (+ 3539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:41.676337 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:41.676340 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:41.676667 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:41.677196 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:41.677201 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:41.677205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:41.677364 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:41.678074 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:41.678078 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:41.678226 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 19:00:41.678834 (+ 608us) client_negotiation.cc:240] Negotiation successful
0924 19:00:41.679137 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8552,"thread_start_us":7955,"threads_started":1}
I20260924 19:00:41.679821 7671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:41.621297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51722 (local address 127.0.97.3:44055)
0924 19:00:41.644887 (+ 23590us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:41.644901 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:41.654413 (+ 9512us) server_negotiation.cc:407] Connection header received
0924 19:00:41.655556 (+ 1143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:41.655566 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:41.655833 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:41.655997 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:41.659536 (+ 3539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:41.661045 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:41.662857 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:41.663751 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:41.663870 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:41.671688 (+ 7818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:41.671705 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:41.671724 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:41.671763 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:41.675547 (+ 3784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:41.676566 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:41.676576 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:41.676580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:41.676743 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:41.677195 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:41.677201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:41.677204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:41.677641 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:41.677919 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:41.679193 (+ 1274us) server_negotiation.cc:299] Negotiation successful
0924 19:00:41.679445 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23376,"thread_start_us":32239,"threads_started":1}
I20260924 19:00:41.680109 7667 heartbeater.cc:347] Connected to a master server at 127.0.97.61:37383
I20260924 19:00:41.680447 7667 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:41.680712 7676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:41.647051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41545 (local address 127.0.97.62:46537)
0924 19:00:41.660014 (+ 12963us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:41.660023 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:41.660043 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:00:41.660097 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:41.660101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:41.660195 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:41.661587 (+ 1392us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:41.661735 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:41.663100 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:41.665723 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:41.666412 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:41.666557 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:41.673005 (+ 6448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:41.673010 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:41.673015 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:41.673071 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:41.676190 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:41.676791 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:41.676796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:41.676801 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:41.677056 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:41.677525 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:41.677531 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:41.677534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:41.677883 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:41.678007 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:41.680166 (+ 2159us) server_negotiation.cc:299] Negotiation successful
0924 19:00:41.680400 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12823,"thread_start_us":13226,"threads_started":1}
I20260924 19:00:41.680858 7665 heartbeater.cc:347] Connected to a master server at 127.0.97.60:43355
I20260924 19:00:41.680987 7665 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:41.681401 7667 heartbeater.cc:511] Master 127.0.97.61:37383 requested a full tablet report, sending...
I20260924 19:00:41.681496 7666 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46537
I20260924 19:00:41.681656 7666 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:41.682063 7666 heartbeater.cc:511] Master 127.0.97.62:46537 requested a full tablet report, sending...
I20260924 19:00:41.682610 7001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:41.681444 7665 heartbeater.cc:511] Master 127.0.97.60:43355 requested a full tablet report, sending...
I20260924 19:00:41.683028 6550 ts_manager.cc:194] Re-registered known tserver with Master: 56fd3560adc24fa5aa71678e700f9836 (127.0.97.2:44407)
I20260924 19:00:41.683251 7148 ts_manager.cc:194] Registered new tserver with Master: 56fd3560adc24fa5aa71678e700f9836 (127.0.97.2:44407)
I20260924 19:00:41.683840 7338 ts_manager.cc:194] Registered new tserver with Master: 56fd3560adc24fa5aa71678e700f9836 (127.0.97.2:44407)
I20260924 19:00:41.684206 6550 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41545
I20260924 19:00:41.776762 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 6878
I20260924 19:00:41.793084 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44055
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46537
--webserver_port=36465
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:46537,127.0.97.61:37383
--tserver_master_addrs=127.0.97.62:46537,127.0.97.61:37383,127.0.97.60:43355 with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:00:42.032629 7518 heartbeater.cc:503] Master 127.0.97.62:46537 was elected leader, sending a full tablet report...
W20260924 19:00:42.123991 7678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:42.124245 7678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:42.124440 7678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:42.131614 7678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:42.131738 7678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:42.131903 7678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:00:42.141844 7678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36465
--tserver_master_addrs=127.0.97.62:46537,127.0.97.61:37383,127.0.97.60:43355
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:42.143162 7678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:42.144593 7678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:42.157871 7684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:42.157848 7683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:42.158200 7686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:42.158488 7678 server_base.cc:1034] running on GCE node
I20260924 19:00:42.159190 7678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:42.160061 7678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:42.161289 7678 hybrid_clock.cc:648] HybridClock initialized: now 1790276442161243 us; error 41 us; skew 500 ppm
I20260924 19:00:42.161684 7678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:42 dist-test-slave-9fn1 krb5kdc[6505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276442, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:42.164804 7678 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260924 19:00:42.166321 7678 webserver.cc:460] Webserver started at http://127.0.97.3:36465/ using document root <none> and password file <none>
I20260924 19:00:42.166983 7678 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:42.167147 7678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:42.172178 7678 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260924 19:00:42.174803 7693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:42.176239 7678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:00:42.176414 7678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "83c576b1596f4264a2d7ce70c7ec7c4a"
format_stamp: "Formatted at 2026-09-24 19:00:38 on dist-test-slave-9fn1"
server_key: "31f263a4edacaac629d28c100cbe4887"
server_key_iv: "57e06f296b1cac9de781eb5dca61c64a"
server_key_version: "encryptionkey@0"
I20260924 19:00:42.177387 7678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:42.198794 7678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:42.202800 7678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:42.203105 7678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:42.204064 7678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:42.205338 7678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:42.205458 7678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:42.205605 7678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:42.205667 7678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:42.237864 7678 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44055
I20260924 19:00:42.237958 7805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44055 every 8 connection(s)
I20260924 19:00:42.239280 7678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:00:42.243199 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 7678
Sep 24 19:00:42 dist-test-slave-9fn1 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276442, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 24 19:00:42 dist-test-slave-9fn1 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276442, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 24 19:00:42 dist-test-slave-9fn1 krb5kdc[6505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276442, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260924 19:00:42.279273 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:42.242799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.97.3:60989)
0924 19:00:42.244063 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:42.244137 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:42.247661 (+ 3524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:42.253803 (+ 6142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:42.253811 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:42.254436 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:42.254788 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:42.254802 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.256626 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.256635 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:42.259496 (+ 2861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:42.259517 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.261579 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.261586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:42.261944 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:42.262779 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:42.262820 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:42.266313 (+ 3493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:42.273277 (+ 6964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:42.273291 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:42.273318 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:42.274036 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:42.274438 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:42.274444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:42.274448 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:42.274726 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:42.275860 (+ 1134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:42.275868 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:42.276742 (+ 874us) client_negotiation.cc:769] Sending connection context
0924 19:00:42.277067 (+ 325us) client_negotiation.cc:240] Negotiation successful
0924 19:00:42.277727 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"mutex_wait_us":4,"thread_start_us":1523,"threads_started":1}
I20260924 19:00:42.282018 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:42.242459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60989 (local address 127.0.97.62:46537)
0924 19:00:42.243617 (+ 1158us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:42.243625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:42.244647 (+ 1022us) server_negotiation.cc:407] Connection header received
0924 19:00:42.248198 (+ 3551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:42.248209 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:42.248323 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:42.248447 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:42.255000 (+ 6553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.256398 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.259694 (+ 3296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.260650 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.260810 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:42.267165 (+ 6355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:42.267174 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:42.267182 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:42.267236 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:42.272127 (+ 4891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:42.274183 (+ 2056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:42.274188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:42.274190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:42.274315 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:42.274842 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:42.274847 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:42.274850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:42.275105 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:42.275262 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:42.281139 (+ 5877us) server_negotiation.cc:299] Negotiation successful
0924 19:00:42.281427 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1018,"thread_start_us":932,"threads_started":1}
I20260924 19:00:42.283401 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:42.244978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43355 (local address 127.0.97.3:54235)
0924 19:00:42.258165 (+ 13187us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:42.258184 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:42.258307 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:42.260771 (+ 2464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:42.260776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:42.261395 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:42.262120 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:42.262131 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.264257 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.264265 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:42.266917 (+ 2652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:42.266931 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.268465 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.268474 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:42.268742 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:42.269329 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:42.269357 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:42.275725 (+ 6368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:42.280585 (+ 4860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:42.280591 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:42.280595 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:42.281112 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:42.281570 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:42.281576 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:42.281580 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:42.281735 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:42.282479 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:42.282483 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:42.282615 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 19:00:42.282742 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 19:00:42.283019 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13049,"mutex_wait_us":326,"thread_start_us":13418,"threads_started":1}
I20260924 19:00:42.283655 7807 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46537
I20260924 19:00:42.283969 7807 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:42.284229 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:42.245059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54235 (local address 127.0.97.60:43355)
0924 19:00:42.260261 (+ 15202us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:42.260267 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:42.260286 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:00:42.260340 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:42.260346 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:42.260430 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:42.260540 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:42.262643 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.264032 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.267182 (+ 3150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.267982 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.268128 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:42.275937 (+ 7809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:42.275946 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:42.275954 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:42.276003 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:42.280429 (+ 4426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:42.281261 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:42.281266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:42.281270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:42.281426 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:42.281882 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:42.281887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:42.281890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:42.282258 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:42.282416 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:42.283687 (+ 1271us) server_negotiation.cc:299] Negotiation successful
0924 19:00:42.283894 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15080,"thread_start_us":15557,"threads_started":1}
I20260924 19:00:42.284766 7807 heartbeater.cc:511] Master 127.0.97.62:46537 requested a full tablet report, sending...
I20260924 19:00:42.285079 7806 heartbeater.cc:347] Connected to a master server at 127.0.97.60:43355
I20260924 19:00:42.285259 7806 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:42.285864 7806 heartbeater.cc:511] Master 127.0.97.60:43355 requested a full tablet report, sending...
I20260924 19:00:42.286896 6550 ts_manager.cc:194] Re-registered known tserver with Master: 83c576b1596f4264a2d7ce70c7ec7c4a (127.0.97.3:44055)
I20260924 19:00:42.288408 6550 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60989
I20260924 19:00:42.289228 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:42.242205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51639 (local address 127.0.97.61:37383)
0924 19:00:42.242706 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:42.242714 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:42.244525 (+ 1811us) server_negotiation.cc:407] Connection header received
0924 19:00:42.248804 (+ 4279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:42.248813 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:42.248919 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:42.249032 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:42.252073 (+ 3041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.253573 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.261455 (+ 7882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.262324 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.262478 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:42.275901 (+ 13423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:42.275908 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:42.275913 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:42.275953 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:42.280428 (+ 4475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:42.286999 (+ 6571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:42.287007 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:42.287011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:42.287196 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:42.287647 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:42.287654 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:42.287658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:42.288010 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:42.288152 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:42.288653 (+ 501us) server_negotiation.cc:299] Negotiation successful
0924 19:00:42.288854 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"spinlock_wait_cycles":9088,"thread_start_us":338,"threads_started":1}
I20260924 19:00:42.289258 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:42.242800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37383 (local address 127.0.97.3:51639)
0924 19:00:42.244066 (+ 1266us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:42.244131 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:42.248612 (+ 4481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:42.249556 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:42.249572 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:42.250763 (+ 1191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:42.251811 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:42.251842 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.259730 (+ 7888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.259739 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:42.261290 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:42.261303 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.262743 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.262753 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:42.263019 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:42.263842 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:42.263882 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:42.275674 (+ 11792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:42.286286 (+ 10612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:42.286292 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:42.286296 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:42.286825 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:42.287359 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:42.287364 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:42.287368 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:42.287515 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:42.288249 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:42.288253 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:42.288401 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 19:00:42.288546 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 19:00:42.288816 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1129,"mutex_wait_us":342,"thread_start_us":413,"threads_started":1}
I20260924 19:00:42.290041 7338 ts_manager.cc:194] Registered new tserver with Master: 83c576b1596f4264a2d7ce70c7ec7c4a (127.0.97.3:44055)
I20260924 19:00:42.290483 7808 heartbeater.cc:347] Connected to a master server at 127.0.97.61:37383
I20260924 19:00:42.290655 7808 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:42.291110 7808 heartbeater.cc:511] Master 127.0.97.61:37383 requested a full tablet report, sending...
I20260924 19:00:42.292734 7148 ts_manager.cc:194] Registered new tserver with Master: 83c576b1596f4264a2d7ce70c7ec7c4a (127.0.97.3:44055)
I20260924 19:00:42.306743 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:42.266914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33964 (local address 127.0.97.62:46537)
0924 19:00:42.282782 (+ 15868us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:42.282787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:42.282802 (+ 15us) server_negotiation.cc:407] Connection header received
0924 19:00:42.282850 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:42.282855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:42.285966 (+ 3111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:42.286101 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:42.289538 (+ 3437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.293239 (+ 3701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.295250 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.295683 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.295820 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:42.299262 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:42.299268 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:42.299272 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:42.299309 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:42.302620 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:42.303972 (+ 1352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:42.303979 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:42.303982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:42.304134 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:42.304553 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:42.304557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:42.304559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:42.304980 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:42.305214 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:42.306120 (+ 906us) server_negotiation.cc:299] Negotiation successful
0924 19:00:42.306356 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15763,"thread_start_us":31529,"threads_started":1}
I20260924 19:00:42.312776 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:42.267250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45066 (local address 127.0.97.61:37383)
0924 19:00:42.289704 (+ 22454us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:42.289709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:42.294353 (+ 4644us) server_negotiation.cc:407] Connection header received
0924 19:00:42.294580 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:42.294586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:42.294655 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:42.294713 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:42.298701 (+ 3988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.300160 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.302141 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.302825 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.302892 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:42.306937 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:42.306944 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:42.306949 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:42.306988 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:42.310075 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:42.310724 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:42.310729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:42.310733 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:42.310867 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:42.311287 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:42.311291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:42.311294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:42.311597 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:42.311830 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:42.312185 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 19:00:42.312413 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22351,"thread_start_us":26404,"threads_started":1}
I20260924 19:00:42.316735 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:42.294630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46286 (local address 127.0.97.60:43355)
0924 19:00:42.294829 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:42.294834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:42.301746 (+ 6912us) server_negotiation.cc:407] Connection header received
0924 19:00:42.301952 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:42.301957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:42.302033 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:42.302136 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:42.303866 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.304918 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.306706 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.307350 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.307485 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:42.310577 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:42.310583 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:42.310588 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:42.310625 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:42.314272 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:42.314972 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:42.314976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:42.314978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:42.315086 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:42.315464 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:42.315470 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:42.315474 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:42.315816 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:42.316030 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:42.316355 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 19:00:42.316499 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 19:00:42.311707 6550 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:00:42.320271 6550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:00:42.349193 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:42.339439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40220 (local address 127.0.97.1:33575)
0924 19:00:42.339960 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:42.339969 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:42.339998 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:00:42.340220 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:42.340231 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:42.340753 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:42.340969 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:42.341906 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.343755 (+ 1849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.346111 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.347809 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.347935 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:42.348149 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:42.348709 (+ 560us) server_negotiation.cc:299] Negotiation successful
0924 19:00:42.348882 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":235,"threads_started":1}
I20260924 19:00:42.349449 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:42.339185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33575 (local address 127.0.0.1:40220)
0924 19:00:42.339888 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:42.339909 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:42.340038 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:42.341205 (+ 1167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:42.341211 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:42.341239 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:42.341674 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:42.341688 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.343922 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.343927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:42.345911 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:42.345928 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.348262 (+ 2334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.348268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:42.348441 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:42.348473 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 19:00:42.348557 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 19:00:42.349070 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":354,"thread_start_us":395,"threads_started":1}
I20260924 19:00:42.349548 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:42.340644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51724 (local address 127.0.97.3:44055)
0924 19:00:42.341245 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:42.341256 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:42.341648 (+ 392us) server_negotiation.cc:407] Connection header received
0924 19:00:42.342177 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:42.342187 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:42.342453 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:42.342639 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:42.343275 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.344389 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.347013 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.348457 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.348587 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:42.348792 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:42.349046 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 19:00:42.349213 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":283,"threads_started":1}
I20260924 19:00:42.350664 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:42.340402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44055 (local address 127.0.0.1:51724)
0924 19:00:42.341543 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:42.341562 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:42.342006 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:42.342747 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:42.342752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:42.342775 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:42.343090 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:42.343099 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.344542 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.344549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:42.346829 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:42.346843 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.348675 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.348682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:42.348863 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:42.348907 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 19:00:42.348990 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 19:00:42.349088 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"thread_start_us":2693,"threads_started":1}
I20260924 19:00:42.352319 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:42.341224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34620 (local address 127.0.97.2:44407)
0924 19:00:42.341847 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:42.341855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:42.344583 (+ 2728us) server_negotiation.cc:407] Connection header received
0924 19:00:42.344679 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:42.344689 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:42.344931 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:42.345111 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:42.346043 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.347302 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.349167 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.351330 (+ 2163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.351493 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:42.351690 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:42.351893 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 19:00:42.352039 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"spinlock_wait_cycles":22912,"thread_start_us":328,"threads_started":1}
I20260924 19:00:42.352362 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:42.340748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44407 (local address 127.0.0.1:34620)
0924 19:00:42.342509 (+ 1761us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:42.342531 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:42.342630 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:42.345298 (+ 2668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:42.345303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:42.345325 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:42.345859 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:42.345872 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.347425 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.347433 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:42.348834 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:42.348845 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.351533 (+ 2688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.351539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:42.351703 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:42.351739 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 19:00:42.351855 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 19:00:42.351991 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1635,"mutex_wait_us":30,"thread_start_us":626,"threads_started":1}
I20260924 19:00:42.353365 7452 tablet_service.cc:1467] Processing CreateTablet for tablet e193da85d6164575a16a3412beceda42 (DEFAULT_TABLE table=test-table [id=885136953c45445eab50a3874f271370]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:42.356189 7452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e193da85d6164575a16a3412beceda42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:42.356408 7741 tablet_service.cc:1467] Processing CreateTablet for tablet e193da85d6164575a16a3412beceda42 (DEFAULT_TABLE table=test-table [id=885136953c45445eab50a3874f271370]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:42.357417 7741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e193da85d6164575a16a3412beceda42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:42.357239 7600 tablet_service.cc:1467] Processing CreateTablet for tablet e193da85d6164575a16a3412beceda42 (DEFAULT_TABLE table=test-table [id=885136953c45445eab50a3874f271370]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:42.358443 7600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e193da85d6164575a16a3412beceda42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:42.369623 7839 tablet_bootstrap.cc:492] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a: Bootstrap starting.
I20260924 19:00:42.372471 7840 tablet_bootstrap.cc:492] T e193da85d6164575a16a3412beceda42 P 31b40f242c874e81a6ab82ce8cb210c7: Bootstrap starting.
I20260924 19:00:42.373775 7839 tablet_bootstrap.cc:654] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:42.375023 7839 log.cc:824] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:42.377180 7840 tablet_bootstrap.cc:654] T e193da85d6164575a16a3412beceda42 P 31b40f242c874e81a6ab82ce8cb210c7: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:42.378804 7840 log.cc:824] T e193da85d6164575a16a3412beceda42 P 31b40f242c874e81a6ab82ce8cb210c7: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:42.384533 7839 tablet_bootstrap.cc:492] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a: No bootstrap required, opened a new log
I20260924 19:00:42.384805 7839 ts_tablet_manager.cc:1397] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.010s
I20260924 19:00:42.387774 7840 tablet_bootstrap.cc:492] T e193da85d6164575a16a3412beceda42 P 31b40f242c874e81a6ab82ce8cb210c7: No bootstrap required, opened a new log
I20260924 19:00:42.388037 7840 ts_tablet_manager.cc:1397] T e193da85d6164575a16a3412beceda42 P 31b40f242c874e81a6ab82ce8cb210c7: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20260924 19:00:42.389901 7841 tablet_bootstrap.cc:492] T e193da85d6164575a16a3412beceda42 P 56fd3560adc24fa5aa71678e700f9836: Bootstrap starting.
I20260924 19:00:42.393343 7841 tablet_bootstrap.cc:654] T e193da85d6164575a16a3412beceda42 P 56fd3560adc24fa5aa71678e700f9836: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:42.394608 7841 log.cc:824] T e193da85d6164575a16a3412beceda42 P 56fd3560adc24fa5aa71678e700f9836: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:42.395009 7839 raft_consensus.cc:348] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b40f242c874e81a6ab82ce8cb210c7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33575 } } peers { permanent_uuid: "83c576b1596f4264a2d7ce70c7ec7c4a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44055 } } peers { permanent_uuid: "56fd3560adc24fa5aa71678e700f9836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44407 } }
I20260924 19:00:42.395689 7839 raft_consensus.cc:374] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:42.395856 7839 raft_consensus.cc:729] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c576b1596f4264a2d7ce70c7ec7c4a, State: Initialized, Role: FOLLOWER
I20260924 19:00:42.396770 7839 consensus_queue.cc:260] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b40f242c874e81a6ab82ce8cb210c7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33575 } } peers { permanent_uuid: "83c576b1596f4264a2d7ce70c7ec7c4a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44055 } } peers { permanent_uuid: "56fd3560adc24fa5aa71678e700f9836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44407 } }
I20260924 19:00:42.398726 7840 raft_consensus.cc:348] T e193da85d6164575a16a3412beceda42 P 31b40f242c874e81a6ab82ce8cb210c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b40f242c874e81a6ab82ce8cb210c7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33575 } } peers { permanent_uuid: "83c576b1596f4264a2d7ce70c7ec7c4a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44055 } } peers { permanent_uuid: "56fd3560adc24fa5aa71678e700f9836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44407 } }
I20260924 19:00:42.399152 7840 raft_consensus.cc:374] T e193da85d6164575a16a3412beceda42 P 31b40f242c874e81a6ab82ce8cb210c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:42.399262 7840 raft_consensus.cc:729] T e193da85d6164575a16a3412beceda42 P 31b40f242c874e81a6ab82ce8cb210c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b40f242c874e81a6ab82ce8cb210c7, State: Initialized, Role: FOLLOWER
I20260924 19:00:42.399839 7840 consensus_queue.cc:260] T e193da85d6164575a16a3412beceda42 P 31b40f242c874e81a6ab82ce8cb210c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b40f242c874e81a6ab82ce8cb210c7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33575 } } peers { permanent_uuid: "83c576b1596f4264a2d7ce70c7ec7c4a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44055 } } peers { permanent_uuid: "56fd3560adc24fa5aa71678e700f9836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44407 } }
I20260924 19:00:42.402347 7840 ts_tablet_manager.cc:1428] T e193da85d6164575a16a3412beceda42 P 31b40f242c874e81a6ab82ce8cb210c7: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260924 19:00:42.402482 7839 ts_tablet_manager.cc:1428] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a: Time spent starting tablet: real 0.018s user 0.005s sys 0.012s
I20260924 19:00:42.407881 7807 heartbeater.cc:503] Master 127.0.97.62:46537 was elected leader, sending a full tablet report...
I20260924 19:00:42.414613 7841 tablet_bootstrap.cc:492] T e193da85d6164575a16a3412beceda42 P 56fd3560adc24fa5aa71678e700f9836: No bootstrap required, opened a new log
I20260924 19:00:42.414947 7841 ts_tablet_manager.cc:1397] T e193da85d6164575a16a3412beceda42 P 56fd3560adc24fa5aa71678e700f9836: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.010s
I20260924 19:00:42.422366 7841 raft_consensus.cc:348] T e193da85d6164575a16a3412beceda42 P 56fd3560adc24fa5aa71678e700f9836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b40f242c874e81a6ab82ce8cb210c7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33575 } } peers { permanent_uuid: "83c576b1596f4264a2d7ce70c7ec7c4a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44055 } } peers { permanent_uuid: "56fd3560adc24fa5aa71678e700f9836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44407 } }
I20260924 19:00:42.422721 7841 raft_consensus.cc:374] T e193da85d6164575a16a3412beceda42 P 56fd3560adc24fa5aa71678e700f9836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:42.422839 7841 raft_consensus.cc:729] T e193da85d6164575a16a3412beceda42 P 56fd3560adc24fa5aa71678e700f9836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56fd3560adc24fa5aa71678e700f9836, State: Initialized, Role: FOLLOWER
I20260924 19:00:42.423470 7841 consensus_queue.cc:260] T e193da85d6164575a16a3412beceda42 P 56fd3560adc24fa5aa71678e700f9836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b40f242c874e81a6ab82ce8cb210c7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33575 } } peers { permanent_uuid: "83c576b1596f4264a2d7ce70c7ec7c4a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44055 } } peers { permanent_uuid: "56fd3560adc24fa5aa71678e700f9836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44407 } }
I20260924 19:00:42.424464 7666 heartbeater.cc:503] Master 127.0.97.62:46537 was elected leader, sending a full tablet report...
I20260924 19:00:42.425879 7841 ts_tablet_manager.cc:1428] T e193da85d6164575a16a3412beceda42 P 56fd3560adc24fa5aa71678e700f9836: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
I20260924 19:00:42.454895 7845 raft_consensus.cc:482] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:00:42.455228 7845 raft_consensus.cc:504] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b40f242c874e81a6ab82ce8cb210c7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33575 } } peers { permanent_uuid: "83c576b1596f4264a2d7ce70c7ec7c4a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44055 } } peers { permanent_uuid: "56fd3560adc24fa5aa71678e700f9836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44407 } }
I20260924 19:00:42.456914 7845 leader_election.cc:288] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31b40f242c874e81a6ab82ce8cb210c7 (127.0.97.1:33575), 56fd3560adc24fa5aa71678e700f9836 (127.0.97.2:44407)
I20260924 19:00:42.463300 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:42.457332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39063 (local address 127.0.97.1:33575)
0924 19:00:42.457583 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:42.457590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:42.457663 (+ 73us) server_negotiation.cc:407] Connection header received
0924 19:00:42.457953 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:42.457961 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:42.458046 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:42.458114 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:42.458920 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.459659 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.461424 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.462287 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.462441 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:42.462533 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:42.462913 (+ 380us) server_negotiation.cc:299] Negotiation successful
0924 19:00:42.463044 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:00:42.463301 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:42.457332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48239 (local address 127.0.97.2:44407)
0924 19:00:42.457558 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:42.457565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:42.457859 (+ 294us) server_negotiation.cc:407] Connection header received
0924 19:00:42.458026 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:42.458031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:42.458112 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:42.458206 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:42.459063 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.459809 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.461449 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.462284 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.462435 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:42.462552 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:42.462919 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 19:00:42.463042 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 19:00:42.463343 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:42.457358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33575 (local address 127.0.97.3:39063)
0924 19:00:42.457563 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:42.457587 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:42.457755 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:42.458159 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:42.458165 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:42.458202 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:42.458435 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:42.458443 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.459791 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.459795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:42.461255 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:42.461266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.462460 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.462466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:42.462742 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:42.462777 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 19:00:42.462886 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 19:00:42.463040 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":104}
I20260924 19:00:42.463421 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:42.457363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44407 (local address 127.0.97.3:48239)
0924 19:00:42.457762 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:42.457776 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:42.457902 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:42.458242 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:42.458247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:42.458276 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:42.458920 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:42.458931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.459970 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.459974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:42.461262 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:42.461273 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.462463 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.462469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:42.462748 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:42.462780 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 19:00:42.462882 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 19:00:42.463034 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":277}
I20260924 19:00:42.464534 7472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e193da85d6164575a16a3412beceda42" candidate_uuid: "83c576b1596f4264a2d7ce70c7ec7c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b40f242c874e81a6ab82ce8cb210c7" is_pre_election: true
I20260924 19:00:42.464560 7620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e193da85d6164575a16a3412beceda42" candidate_uuid: "83c576b1596f4264a2d7ce70c7ec7c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56fd3560adc24fa5aa71678e700f9836" is_pre_election: true
I20260924 19:00:42.465123 7620 raft_consensus.cc:2454] T e193da85d6164575a16a3412beceda42 P 56fd3560adc24fa5aa71678e700f9836 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c576b1596f4264a2d7ce70c7ec7c4a in term 0.
I20260924 19:00:42.465123 7472 raft_consensus.cc:2454] T e193da85d6164575a16a3412beceda42 P 31b40f242c874e81a6ab82ce8cb210c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c576b1596f4264a2d7ce70c7ec7c4a in term 0.
I20260924 19:00:42.465854 7696 leader_election.cc:302] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b40f242c874e81a6ab82ce8cb210c7, 83c576b1596f4264a2d7ce70c7ec7c4a; no voters:
I20260924 19:00:42.466279 7845 raft_consensus.cc:2781] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:00:42.466434 7845 raft_consensus.cc:482] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:00:42.466544 7845 raft_consensus.cc:3037] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:42.468775 7845 raft_consensus.cc:504] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b40f242c874e81a6ab82ce8cb210c7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33575 } } peers { permanent_uuid: "83c576b1596f4264a2d7ce70c7ec7c4a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44055 } } peers { permanent_uuid: "56fd3560adc24fa5aa71678e700f9836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44407 } }
I20260924 19:00:42.469373 7845 leader_election.cc:288] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a [CANDIDATE]: Term 1 election: Requested vote from peers 31b40f242c874e81a6ab82ce8cb210c7 (127.0.97.1:33575), 56fd3560adc24fa5aa71678e700f9836 (127.0.97.2:44407)
I20260924 19:00:42.469841 7472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e193da85d6164575a16a3412beceda42" candidate_uuid: "83c576b1596f4264a2d7ce70c7ec7c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b40f242c874e81a6ab82ce8cb210c7"
I20260924 19:00:42.469925 7620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e193da85d6164575a16a3412beceda42" candidate_uuid: "83c576b1596f4264a2d7ce70c7ec7c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56fd3560adc24fa5aa71678e700f9836"
I20260924 19:00:42.470052 7472 raft_consensus.cc:3037] T e193da85d6164575a16a3412beceda42 P 31b40f242c874e81a6ab82ce8cb210c7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:42.470180 7620 raft_consensus.cc:3037] T e193da85d6164575a16a3412beceda42 P 56fd3560adc24fa5aa71678e700f9836 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:42.472610 7620 raft_consensus.cc:2454] T e193da85d6164575a16a3412beceda42 P 56fd3560adc24fa5aa71678e700f9836 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c576b1596f4264a2d7ce70c7ec7c4a in term 1.
I20260924 19:00:42.472620 7472 raft_consensus.cc:2454] T e193da85d6164575a16a3412beceda42 P 31b40f242c874e81a6ab82ce8cb210c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c576b1596f4264a2d7ce70c7ec7c4a in term 1.
I20260924 19:00:42.473042 7697 leader_election.cc:302] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56fd3560adc24fa5aa71678e700f9836, 83c576b1596f4264a2d7ce70c7ec7c4a; no voters:
I20260924 19:00:42.473313 7845 raft_consensus.cc:2781] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:00:42.473870 7845 raft_consensus.cc:686] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a [term 1 LEADER]: Becoming Leader. State: Replica: 83c576b1596f4264a2d7ce70c7ec7c4a, State: Running, Role: LEADER
I20260924 19:00:42.474454 7845 consensus_queue.cc:237] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b40f242c874e81a6ab82ce8cb210c7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33575 } } peers { permanent_uuid: "83c576b1596f4264a2d7ce70c7ec7c4a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44055 } } peers { permanent_uuid: "56fd3560adc24fa5aa71678e700f9836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44407 } }
I20260924 19:00:42.479497 6550 catalog_manager.cc:5495] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 83c576b1596f4264a2d7ce70c7ec7c4a (127.0.97.3). New cstate: current_term: 1 leader_uuid: "83c576b1596f4264a2d7ce70c7ec7c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31b40f242c874e81a6ab82ce8cb210c7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83c576b1596f4264a2d7ce70c7ec7c4a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56fd3560adc24fa5aa71678e700f9836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44407 } health_report { overall_health: UNKNOWN } } }
W20260924 19:00:42.493579 7809 tablet.cc:2369] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:00:42.496109 7520 tablet.cc:2369] T e193da85d6164575a16a3412beceda42 P 31b40f242c874e81a6ab82ce8cb210c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:00:42.515357 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:42.510400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51736 (local address 127.0.97.3:44055)
0924 19:00:42.510573 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:42.510577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:42.510594 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:00:42.510805 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:42.510809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:42.510899 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:42.511016 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:00:42.511330 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.512011 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.513277 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.513682 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.513790 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:42.514289 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:00:42.514789 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:00:42.514924 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:42.514967 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 19:00:42.515027 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 19:00:42.523207 7620 raft_consensus.cc:1261] T e193da85d6164575a16a3412beceda42 P 56fd3560adc24fa5aa71678e700f9836 [term 1 FOLLOWER]: Refusing update from remote peer 83c576b1596f4264a2d7ce70c7ec7c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:00:42.523286 7472 raft_consensus.cc:1261] T e193da85d6164575a16a3412beceda42 P 31b40f242c874e81a6ab82ce8cb210c7 [term 1 FOLLOWER]: Refusing update from remote peer 83c576b1596f4264a2d7ce70c7ec7c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:00:42.524171 7845 consensus_queue.cc:1035] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "56fd3560adc24fa5aa71678e700f9836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:00:42.524447 7848 consensus_queue.cc:1035] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b40f242c874e81a6ab82ce8cb210c7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:00:42.536859 7851 mvcc.cc:205] Tried to move back new op lower bound from 7332972308565753856 to 7332972308378624000. Current Snapshot: MvccSnapshot[applied={T|T < 7332972308565753856}]
I20260924 19:00:42.542542 7852 mvcc.cc:205] Tried to move back new op lower bound from 7332972308565753856 to 7332972308378624000. Current Snapshot: MvccSnapshot[applied={T|T < 7332972308565753856}]
I20260924 19:00:42.562626 6550 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:00:42.562891 6550 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:00:42.569042 6550 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f30647f1f9c43cf929f429a1e49e814: Sending DeleteTablet for 3 replicas of tablet e193da85d6164575a16a3412beceda42
I20260924 19:00:42.570314 7452 tablet_service.cc:1514] Processing DeleteTablet for tablet e193da85d6164575a16a3412beceda42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:42 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40220
I20260924 19:00:42.570683 7600 tablet_service.cc:1514] Processing DeleteTablet for tablet e193da85d6164575a16a3412beceda42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:42 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34620
I20260924 19:00:42.571256 7861 tablet_replica.cc:330] T e193da85d6164575a16a3412beceda42 P 31b40f242c874e81a6ab82ce8cb210c7: stopping tablet replica
I20260924 19:00:42.571769 7861 raft_consensus.cc:2229] T e193da85d6164575a16a3412beceda42 P 31b40f242c874e81a6ab82ce8cb210c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:00:42.572321 7861 raft_consensus.cc:2258] T e193da85d6164575a16a3412beceda42 P 31b40f242c874e81a6ab82ce8cb210c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:42.573132 7741 tablet_service.cc:1514] Processing DeleteTablet for tablet e193da85d6164575a16a3412beceda42 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:42 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51724
I20260924 19:00:42.575321 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 6518
I20260924 19:00:42.576393 6580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:42.577030 7862 tablet_replica.cc:330] T e193da85d6164575a16a3412beceda42 P 56fd3560adc24fa5aa71678e700f9836: stopping tablet replica
I20260924 19:00:42.577308 7864 tablet_replica.cc:330] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a: stopping tablet replica
I20260924 19:00:42.577814 7862 raft_consensus.cc:2229] T e193da85d6164575a16a3412beceda42 P 56fd3560adc24fa5aa71678e700f9836 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:00:42.577865 7864 raft_consensus.cc:2229] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:00:42.578349 7862 raft_consensus.cc:2258] T e193da85d6164575a16a3412beceda42 P 56fd3560adc24fa5aa71678e700f9836 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:42.578502 7864 raft_consensus.cc:2258] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:42.584911 7861 ts_tablet_manager.cc:1905] T e193da85d6164575a16a3412beceda42 P 31b40f242c874e81a6ab82ce8cb210c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:42.592621 7864 ts_tablet_manager.cc:1905] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:42.597649 7862 ts_tablet_manager.cc:1905] T e193da85d6164575a16a3412beceda42 P 56fd3560adc24fa5aa71678e700f9836: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:42.598057 7861 ts_tablet_manager.cc:1918] T e193da85d6164575a16a3412beceda42 P 31b40f242c874e81a6ab82ce8cb210c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:00:42.598232 7861 log.cc:1196] T e193da85d6164575a16a3412beceda42 P 31b40f242c874e81a6ab82ce8cb210c7: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/wal/wals/e193da85d6164575a16a3412beceda42
I20260924 19:00:42.598757 7861 ts_tablet_manager.cc:1939] T e193da85d6164575a16a3412beceda42 P 31b40f242c874e81a6ab82ce8cb210c7: Deleting consensus metadata
I20260924 19:00:42.604297 7864 ts_tablet_manager.cc:1918] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:00:42.604468 7864 log.cc:1196] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/wal/wals/e193da85d6164575a16a3412beceda42
I20260924 19:00:42.604913 7864 ts_tablet_manager.cc:1939] T e193da85d6164575a16a3412beceda42 P 83c576b1596f4264a2d7ce70c7ec7c4a: Deleting consensus metadata
I20260924 19:00:42.609583 7862 ts_tablet_manager.cc:1918] T e193da85d6164575a16a3412beceda42 P 56fd3560adc24fa5aa71678e700f9836: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:00:42.609835 7862 log.cc:1196] T e193da85d6164575a16a3412beceda42 P 56fd3560adc24fa5aa71678e700f9836: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/wal/wals/e193da85d6164575a16a3412beceda42
I20260924 19:00:42.610450 7862 ts_tablet_manager.cc:1939] T e193da85d6164575a16a3412beceda42 P 56fd3560adc24fa5aa71678e700f9836: Deleting consensus metadata
I20260924 19:00:42.617491 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:42.573236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45080 (local address 127.0.97.61:37383)
0924 19:00:42.573460 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:42.573467 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:42.573655 (+ 188us) server_negotiation.cc:407] Connection header received
0924 19:00:42.573902 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:42.573909 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:42.573999 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:42.574090 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:42.578557 (+ 4467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.590932 (+ 12375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.595851 (+ 4919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.596496 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.596610 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:42.609471 (+ 12861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:42.609478 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:42.609481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:42.609583 (+ 102us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:42.614139 (+ 4556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:42.615435 (+ 1296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:42.615440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:42.615443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:42.615575 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:42.615965 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:42.615971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:42.615974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:42.616301 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:42.616570 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:42.616933 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 19:00:42.617138 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260924 19:00:42.630076 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:42.574094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46298 (local address 127.0.97.60:43355)
0924 19:00:42.574256 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:42.574262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:42.618063 (+ 43801us) server_negotiation.cc:407] Connection header received
0924 19:00:42.618311 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:42.618316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:42.618393 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:42.618466 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:42.619898 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.620754 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.622084 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.622476 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.622621 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:42.624969 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:42.624974 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:42.624977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:42.625007 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:42.627516 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:42.628148 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:42.628160 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:42.628164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:42.628285 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:42.628650 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:42.628656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:42.628660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:42.628925 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:42.629168 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:42.629469 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 19:00:42.629717 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 19:00:42.767555 6534 catalog_manager.cc:4841] TS 83c576b1596f4264a2d7ce70c7ec7c4a (127.0.97.3:44055): tablet e193da85d6164575a16a3412beceda42 (table test-table [id=885136953c45445eab50a3874f271370]) successfully deleted
I20260924 19:00:42.767673 6536 catalog_manager.cc:4841] TS 31b40f242c874e81a6ab82ce8cb210c7 (127.0.97.1:33575): tablet e193da85d6164575a16a3412beceda42 (table test-table [id=885136953c45445eab50a3874f271370]) successfully deleted
I20260924 19:00:42.767884 6537 catalog_manager.cc:4841] TS 56fd3560adc24fa5aa71678e700f9836 (127.0.97.2:44407): tablet e193da85d6164575a16a3412beceda42 (table test-table [id=885136953c45445eab50a3874f271370]) successfully deleted
I20260924 19:00:42.768374 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 6518
I20260924 19:00:42.825343 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:42.805075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45096 (local address 127.0.97.61:37383)
0924 19:00:42.805297 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:42.805306 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:42.806318 (+ 1012us) server_negotiation.cc:407] Connection header received
0924 19:00:42.806646 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:42.806656 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:42.806759 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:42.806860 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:42.809720 (+ 2860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.811164 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.813730 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.814639 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.814762 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:42.819229 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:42.819238 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:42.819246 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:42.819289 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:42.822245 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:42.823167 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:42.823173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:42.823177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:42.823326 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:42.823763 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:42.823769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:42.823773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:42.824154 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:42.824410 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:42.824749 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 19:00:42.824976 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 19:00:42.830086 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:42.806369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46310 (local address 127.0.97.60:43355)
0924 19:00:42.806582 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:42.806590 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:42.816258 (+ 9668us) server_negotiation.cc:407] Connection header received
0924 19:00:42.816505 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:42.816514 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:42.816620 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:42.816705 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:42.818380 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.819166 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.820934 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:42.821663 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:42.821793 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:42.824660 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:42.824666 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:42.824671 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:42.824706 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:42.827557 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:42.828188 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:42.828195 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:42.828199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:42.828330 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:42.828691 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:42.828697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:42.828700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:42.828975 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:42.829219 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:42.829558 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 19:00:42.829769 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 19:00:43.548552 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:43.547634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.97.3:41869)
0924 19:00:43.548037 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:43.548270 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":207,"threads_started":1}
W20260924 19:00:43.549032 7807 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46537 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
I20260924 19:00:43.553272 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:43.552655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.97.1:36179)
0924 19:00:43.553016 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:43.553129 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":187,"threads_started":1}
W20260924 19:00:43.553666 7518 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46537 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
I20260924 19:00:43.558626 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:43.558056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.97.2:41115)
0924 19:00:43.558399 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:43.558487 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":176,"threads_started":1}
W20260924 19:00:43.559012 7666 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46537 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
I20260924 19:00:44.070026 7876 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f30647f1f9c43cf929f429a1e49e814)
I20260924 19:00:44.070266 7876 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: false } }
I20260924 19:00:44.071753 7876 leader_election.cc:288] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f30647f1f9c43cf929f429a1e49e814 (127.0.97.62:46537), 3c7e501be5fb40ab92a39564889cadc9 (127.0.97.60:43355)
I20260924 19:00:44.072800 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.071805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.0.1:34368)
0924 19:00:44.072307 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.072515 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":212,"threads_started":1}
I20260924 19:00:44.072873 7878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8f30647f1f9c43cf929f429a1e49e814)
I20260924 19:00:44.073246 7878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: false } }
W20260924 19:00:44.074080 7133 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
I20260924 19:00:44.075063 7878 leader_election.cc:288] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8f30647f1f9c43cf929f429a1e49e814 (127.0.97.62:46537), 07bd526e5ab649bb873f289aee41b508 (127.0.97.61:37383)
I20260924 19:00:44.075982 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.075353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.0.1:34388)
0924 19:00:44.075518 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.075725 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 19:00:44.076161 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.075214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.0.1:34372)
0924 19:00:44.075731 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.075889 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":278,"threads_started":1}
W20260924 19:00:44.076457 7133 leader_election.cc:334] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8f30647f1f9c43cf929f429a1e49e814 (127.0.97.62:46537): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
W20260924 19:00:44.076515 7322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
I20260924 19:00:44.078929 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.078075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.0.1:34394)
0924 19:00:44.078543 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.078659 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":262,"threads_started":1}
W20260924 19:00:44.079082 7322 leader_election.cc:334] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8f30647f1f9c43cf929f429a1e49e814 (127.0.97.62:46537): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
I20260924 19:00:44.082038 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.072767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46312 (local address 127.0.97.60:43355)
0924 19:00:44.073854 (+ 1087us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:44.073862 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:44.073882 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:00:44.073963 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:44.073967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:44.074018 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:44.074140 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:44.074616 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.077474 (+ 2858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.079874 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.081239 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.081350 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:44.081432 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:44.081715 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 19:00:44.081830 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":947,"thread_start_us":817,"threads_started":1}
I20260924 19:00:44.082186 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.072393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:43355 (local address 127.0.0.1:46312)
0924 19:00:44.073080 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.073103 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:44.073268 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:44.074159 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:44.074164 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:44.074192 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:44.074461 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:44.074471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.077656 (+ 3185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.077665 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:44.079748 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:44.079758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.081370 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.081374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:44.081568 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:44.081592 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 19:00:44.081664 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 19:00:44.081808 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":2,"thread_start_us":1396,"threads_started":1}
I20260924 19:00:44.082921 7348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07bd526e5ab649bb873f289aee41b508" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3c7e501be5fb40ab92a39564889cadc9" is_pre_election: true
I20260924 19:00:44.083240 7348 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07bd526e5ab649bb873f289aee41b508 in term 1.
I20260924 19:00:44.083750 7133 leader_election.cc:302] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07bd526e5ab649bb873f289aee41b508, 3c7e501be5fb40ab92a39564889cadc9; no voters: 8f30647f1f9c43cf929f429a1e49e814
I20260924 19:00:44.084069 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.076559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:37383 (local address 127.0.0.1:45104)
0924 19:00:44.076842 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.076861 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:44.076990 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:44.077548 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:44.077555 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:44.077588 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:44.077874 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:44.077884 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.079527 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.079531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:44.081613 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:44.081623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.083393 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.083403 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:44.083597 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:44.083630 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 19:00:44.083755 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 19:00:44.083848 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":2}
I20260924 19:00:44.084059 7876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 19:00:44.084172 7876 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8f30647f1f9c43cf929f429a1e49e814)
I20260924 19:00:44.084285 7876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:00:44.084646 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.075296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45104 (local address 127.0.97.61:37383)
0924 19:00:44.077200 (+ 1904us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:44.077208 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:44.077235 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:00:44.077291 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:44.077296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:44.077377 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:44.077486 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:44.078066 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.079391 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.081774 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.083134 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.083268 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:44.083374 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:44.084251 (+ 877us) server_negotiation.cc:299] Negotiation successful
0924 19:00:44.084395 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":1771,"thread_start_us":1615,"threads_started":1}
I20260924 19:00:44.085196 7157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3c7e501be5fb40ab92a39564889cadc9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "07bd526e5ab649bb873f289aee41b508" is_pre_election: true
I20260924 19:00:44.086542 7876 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: false } }
I20260924 19:00:44.086948 7157 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c7e501be5fb40ab92a39564889cadc9 in current term 2: Already voted for candidate 07bd526e5ab649bb873f289aee41b508 in this term.
I20260924 19:00:44.087199 7876 leader_election.cc:288] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [CANDIDATE]: Term 2 election: Requested vote from peers 8f30647f1f9c43cf929f429a1e49e814 (127.0.97.62:46537), 3c7e501be5fb40ab92a39564889cadc9 (127.0.97.60:43355)
I20260924 19:00:44.087661 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.087242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.0.1:34398)
0924 19:00:44.087433 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.087517 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260924 19:00:44.087767 7348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07bd526e5ab649bb873f289aee41b508" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3c7e501be5fb40ab92a39564889cadc9"
I20260924 19:00:44.087950 7321 leader_election.cc:302] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c7e501be5fb40ab92a39564889cadc9; no voters: 07bd526e5ab649bb873f289aee41b508, 8f30647f1f9c43cf929f429a1e49e814
I20260924 19:00:44.088001 7348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:00:44.088634 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.088345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.0.1:34404)
0924 19:00:44.088469 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.088548 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 19:00:44.088837 7133 leader_election.cc:334] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8f30647f1f9c43cf929f429a1e49e814 (127.0.97.62:46537): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
I20260924 19:00:44.090376 7348 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07bd526e5ab649bb873f289aee41b508 in term 2.
I20260924 19:00:44.090619 7878 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:00:44.090857 7133 leader_election.cc:302] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07bd526e5ab649bb873f289aee41b508, 3c7e501be5fb40ab92a39564889cadc9; no voters: 8f30647f1f9c43cf929f429a1e49e814
I20260924 19:00:44.091239 7876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:00:44.091639 7876 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [term 2 LEADER]: Becoming Leader. State: Replica: 07bd526e5ab649bb873f289aee41b508, State: Running, Role: LEADER
I20260924 19:00:44.092008 7876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: false } }
I20260924 19:00:44.093727 7886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07bd526e5ab649bb873f289aee41b508. Latest consensus state: current_term: 2 leader_uuid: "07bd526e5ab649bb873f289aee41b508" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: false } } }
I20260924 19:00:44.094143 7886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:44.094991 7888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:00:44.098335 7888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:00:44.099045 7888 catalog_manager.cc:1259] Loaded cluster ID: d0f1f7a463e442e28820eee13444c245
I20260924 19:00:44.099110 7888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:00:44.100113 7888 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:00:44.100833 7888 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508: Loaded TSK: 0
I20260924 19:00:44.101675 7888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:00:44.112722 7148 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:00:44.119369 7148 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:00:44.123114 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.122714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.0.1:34424)
0924 19:00:44.122854 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.122956 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 19:00:44.123435 7348 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [term 2 FOLLOWER]: Refusing update from remote peer 07bd526e5ab649bb873f289aee41b508: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260924 19:00:44.124019 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.123665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.0.1:34434)
0924 19:00:44.123815 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.123886 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 19:00:44.124058 7886 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260924 19:00:44.124471 7133 consensus_peers.cc:487] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 -> Peer 8f30647f1f9c43cf929f429a1e49e814 (127.0.97.62:46537): Couldn't send request to peer 8f30647f1f9c43cf929f429a1e49e814. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 19:00:44.126855 7878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07bd526e5ab649bb873f289aee41b508. Latest consensus state: current_term: 2 leader_uuid: "07bd526e5ab649bb873f289aee41b508" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: false } } }
I20260924 19:00:44.127107 7878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:00:44.128366 7886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "07bd526e5ab649bb873f289aee41b508" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: false } } }
I20260924 19:00:44.128625 7886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508 [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:44.129117 7878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "07bd526e5ab649bb873f289aee41b508" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "8f30647f1f9c43cf929f429a1e49e814" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46537 } } peers { permanent_uuid: "07bd526e5ab649bb873f289aee41b508" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 37383 } attrs { promote: false } } peers { permanent_uuid: "3c7e501be5fb40ab92a39564889cadc9" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 43355 } attrs { promote: false } } }
I20260924 19:00:44.129808 7878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7e501be5fb40ab92a39564889cadc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:00:44.145566 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.136828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44407 (local address 127.0.0.1:34632)
0924 19:00:44.136992 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.137009 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:44.137123 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:44.138883 (+ 1760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:44.138890 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:44.138934 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:44.139303 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:44.139314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.140762 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.140769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:44.143010 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:44.143024 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.144696 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.144703 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:44.144867 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:44.144905 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 19:00:44.144994 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 19:00:44.145134 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":58}
I20260924 19:00:44.146654 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.136923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34632 (local address 127.0.97.2:44407)
0924 19:00:44.138057 (+ 1134us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:44.138066 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:44.138090 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:00:44.138649 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:44.138655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:44.138729 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:44.139436 (+ 707us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:44.139600 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.140640 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.143185 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.144459 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.146088 (+ 1629us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:44.146177 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:44.146228 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 19:00:44.146357 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":982,"thread_start_us":848,"threads_started":1}
I20260924 19:00:44.147490 7600 tablet_service.cc:1467] Processing CreateTablet for tablet 514b90182c4541b0842ac294e9193455 (DEFAULT_TABLE table=test-table [id=0939c3b6892d46cba8c963b13c56163b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:44.147737 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.137854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33575 (local address 127.0.0.1:40224)
0924 19:00:44.138381 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.138394 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:44.138524 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:44.139141 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:44.139150 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:44.139174 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:44.139635 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:44.139649 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.140632 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.140639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:44.143967 (+ 3328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:44.143980 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.145738 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.145746 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:44.145906 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:44.145937 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 19:00:44.146019 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 19:00:44.146146 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":84,"thread_start_us":237,"threads_started":1}
I20260924 19:00:44.147955 7600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 514b90182c4541b0842ac294e9193455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:44.148373 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.138269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40224 (local address 127.0.97.1:33575)
0924 19:00:44.138714 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:44.138719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:44.138731 (+ 12us) server_negotiation.cc:407] Connection header received
0924 19:00:44.138812 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:44.138816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:44.138887 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:44.138988 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:44.139816 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.140510 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.144137 (+ 3627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.145325 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.145437 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:44.145580 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:44.147964 (+ 2384us) server_negotiation.cc:299] Negotiation successful
0924 19:00:44.148092 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":222,"threads_started":1}
I20260924 19:00:44.149034 7452 tablet_service.cc:1467] Processing CreateTablet for tablet 514b90182c4541b0842ac294e9193455 (DEFAULT_TABLE table=test-table [id=0939c3b6892d46cba8c963b13c56163b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:44.149425 7452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 514b90182c4541b0842ac294e9193455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:44.160820 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.135860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44055 (local address 127.0.0.1:51742)
0924 19:00:44.136034 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.136052 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:44.136151 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:44.154541 (+ 18390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:44.154547 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:44.154574 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:44.154750 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:44.154761 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.156604 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.156610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:44.158542 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:44.158555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.160122 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.160128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:44.160325 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:44.160360 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 19:00:44.160417 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 19:00:44.160551 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 19:00:44.161682 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.135956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51742 (local address 127.0.97.3:44055)
0924 19:00:44.154136 (+ 18180us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:44.154142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:44.154161 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:00:44.154229 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:44.154234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:44.154303 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:44.154880 (+ 577us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:44.155033 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.156427 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.158753 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.159958 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.161132 (+ 1174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:44.161229 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:44.161274 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 19:00:44.161392 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18056,"thread_start_us":18441,"threads_started":1}
I20260924 19:00:44.161993 7899 tablet_bootstrap.cc:492] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836: Bootstrap starting.
I20260924 19:00:44.162408 7741 tablet_service.cc:1467] Processing CreateTablet for tablet 514b90182c4541b0842ac294e9193455 (DEFAULT_TABLE table=test-table [id=0939c3b6892d46cba8c963b13c56163b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:44.162803 7741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 514b90182c4541b0842ac294e9193455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:44.163818 7899 tablet_bootstrap.cc:654] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:44.166106 7900 tablet_bootstrap.cc:492] T 514b90182c4541b0842ac294e9193455 P 31b40f242c874e81a6ab82ce8cb210c7: Bootstrap starting.
I20260924 19:00:44.167562 7900 tablet_bootstrap.cc:654] T 514b90182c4541b0842ac294e9193455 P 31b40f242c874e81a6ab82ce8cb210c7: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:44.169899 7900 tablet_bootstrap.cc:492] T 514b90182c4541b0842ac294e9193455 P 31b40f242c874e81a6ab82ce8cb210c7: No bootstrap required, opened a new log
I20260924 19:00:44.170037 7903 tablet_bootstrap.cc:492] T 514b90182c4541b0842ac294e9193455 P 83c576b1596f4264a2d7ce70c7ec7c4a: Bootstrap starting.
I20260924 19:00:44.170037 7900 ts_tablet_manager.cc:1397] T 514b90182c4541b0842ac294e9193455 P 31b40f242c874e81a6ab82ce8cb210c7: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260924 19:00:44.170660 7899 tablet_bootstrap.cc:492] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836: No bootstrap required, opened a new log
I20260924 19:00:44.170794 7899 ts_tablet_manager.cc:1397] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260924 19:00:44.170841 7900 raft_consensus.cc:348] T 514b90182c4541b0842ac294e9193455 P 31b40f242c874e81a6ab82ce8cb210c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c576b1596f4264a2d7ce70c7ec7c4a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44055 } } peers { permanent_uuid: "56fd3560adc24fa5aa71678e700f9836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44407 } } peers { permanent_uuid: "31b40f242c874e81a6ab82ce8cb210c7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33575 } }
I20260924 19:00:44.171111 7900 raft_consensus.cc:374] T 514b90182c4541b0842ac294e9193455 P 31b40f242c874e81a6ab82ce8cb210c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:44.171203 7900 raft_consensus.cc:729] T 514b90182c4541b0842ac294e9193455 P 31b40f242c874e81a6ab82ce8cb210c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b40f242c874e81a6ab82ce8cb210c7, State: Initialized, Role: FOLLOWER
I20260924 19:00:44.171491 7900 consensus_queue.cc:260] T 514b90182c4541b0842ac294e9193455 P 31b40f242c874e81a6ab82ce8cb210c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c576b1596f4264a2d7ce70c7ec7c4a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44055 } } peers { permanent_uuid: "56fd3560adc24fa5aa71678e700f9836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44407 } } peers { permanent_uuid: "31b40f242c874e81a6ab82ce8cb210c7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33575 } }
I20260924 19:00:44.171581 7899 raft_consensus.cc:348] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c576b1596f4264a2d7ce70c7ec7c4a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44055 } } peers { permanent_uuid: "56fd3560adc24fa5aa71678e700f9836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44407 } } peers { permanent_uuid: "31b40f242c874e81a6ab82ce8cb210c7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33575 } }
I20260924 19:00:44.171657 7903 tablet_bootstrap.cc:654] T 514b90182c4541b0842ac294e9193455 P 83c576b1596f4264a2d7ce70c7ec7c4a: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:44.171787 7899 raft_consensus.cc:374] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:44.171914 7899 raft_consensus.cc:729] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56fd3560adc24fa5aa71678e700f9836, State: Initialized, Role: FOLLOWER
I20260924 19:00:44.172191 7899 consensus_queue.cc:260] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c576b1596f4264a2d7ce70c7ec7c4a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44055 } } peers { permanent_uuid: "56fd3560adc24fa5aa71678e700f9836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44407 } } peers { permanent_uuid: "31b40f242c874e81a6ab82ce8cb210c7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33575 } }
I20260924 19:00:44.172348 7900 ts_tablet_manager.cc:1428] T 514b90182c4541b0842ac294e9193455 P 31b40f242c874e81a6ab82ce8cb210c7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:00:44.174716 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.173000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.97.1:33619)
0924 19:00:44.174188 (+ 1188us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.174379 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1011,"threads_started":1}
I20260924 19:00:44.177953 7519 heartbeater.cc:503] Master 127.0.97.61:37383 was elected leader, sending a full tablet report...
I20260924 19:00:44.179802 7899 ts_tablet_manager.cc:1428] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836: Time spent starting tablet: real 0.008s user 0.003s sys 0.006s
I20260924 19:00:44.180364 7903 tablet_bootstrap.cc:492] T 514b90182c4541b0842ac294e9193455 P 83c576b1596f4264a2d7ce70c7ec7c4a: No bootstrap required, opened a new log
I20260924 19:00:44.180500 7903 ts_tablet_manager.cc:1397] T 514b90182c4541b0842ac294e9193455 P 83c576b1596f4264a2d7ce70c7ec7c4a: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260924 19:00:44.181210 7903 raft_consensus.cc:348] T 514b90182c4541b0842ac294e9193455 P 83c576b1596f4264a2d7ce70c7ec7c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c576b1596f4264a2d7ce70c7ec7c4a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44055 } } peers { permanent_uuid: "56fd3560adc24fa5aa71678e700f9836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44407 } } peers { permanent_uuid: "31b40f242c874e81a6ab82ce8cb210c7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33575 } }
I20260924 19:00:44.181450 7903 raft_consensus.cc:374] T 514b90182c4541b0842ac294e9193455 P 83c576b1596f4264a2d7ce70c7ec7c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:44.181573 7903 raft_consensus.cc:729] T 514b90182c4541b0842ac294e9193455 P 83c576b1596f4264a2d7ce70c7ec7c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c576b1596f4264a2d7ce70c7ec7c4a, State: Initialized, Role: FOLLOWER
I20260924 19:00:44.181806 7903 consensus_queue.cc:260] T 514b90182c4541b0842ac294e9193455 P 83c576b1596f4264a2d7ce70c7ec7c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c576b1596f4264a2d7ce70c7ec7c4a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44055 } } peers { permanent_uuid: "56fd3560adc24fa5aa71678e700f9836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44407 } } peers { permanent_uuid: "31b40f242c874e81a6ab82ce8cb210c7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33575 } }
I20260924 19:00:44.183763 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.177089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.97.1:58371)
0924 19:00:44.183314 (+ 6225us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.183579 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6085}
I20260924 19:00:44.185729 7667 heartbeater.cc:503] Master 127.0.97.61:37383 was elected leader, sending a full tablet report...
I20260924 19:00:44.186786 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.180328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.97.2:46931)
0924 19:00:44.186269 (+ 5941us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.186449 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5770,"thread_start_us":5564,"threads_started":1}
I20260924 19:00:44.188336 7903 ts_tablet_manager.cc:1428] T 514b90182c4541b0842ac294e9193455 P 83c576b1596f4264a2d7ce70c7ec7c4a: Time spent starting tablet: real 0.008s user 0.000s sys 0.008s
I20260924 19:00:44.189121 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.187961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.97.2:54603)
0924 19:00:44.188906 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.188995 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":838}
I20260924 19:00:44.189327 7808 heartbeater.cc:503] Master 127.0.97.61:37383 was elected leader, sending a full tablet report...
I20260924 19:00:44.190332 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.187713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.97.3:50933)
0924 19:00:44.189853 (+ 2140us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.190015 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2007,"spinlock_wait_cycles":7040,"thread_start_us":1994,"threads_started":1}
I20260924 19:00:44.191229 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.190893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.97.3:35479)
0924 19:00:44.191019 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.191094 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 19:00:44.229565 7905 raft_consensus.cc:482] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:00:44.229844 7905 raft_consensus.cc:504] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c576b1596f4264a2d7ce70c7ec7c4a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44055 } } peers { permanent_uuid: "56fd3560adc24fa5aa71678e700f9836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44407 } } peers { permanent_uuid: "31b40f242c874e81a6ab82ce8cb210c7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33575 } }
I20260924 19:00:44.231317 7905 leader_election.cc:288] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83c576b1596f4264a2d7ce70c7ec7c4a (127.0.97.3:44055), 31b40f242c874e81a6ab82ce8cb210c7 (127.0.97.1:33575)
I20260924 19:00:44.239315 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.231705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45633 (local address 127.0.97.1:33575)
0924 19:00:44.231899 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:44.231907 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:44.232458 (+ 551us) server_negotiation.cc:407] Connection header received
0924 19:00:44.232571 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:44.232579 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:44.232658 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:44.232753 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:44.233286 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.234123 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.236524 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.237785 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.237934 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:44.238037 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:44.238938 (+ 901us) server_negotiation.cc:299] Negotiation successful
0924 19:00:44.239073 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 19:00:44.239302 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.231576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44055 (local address 127.0.97.2:44205)
0924 19:00:44.231852 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.231874 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:44.232076 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:44.232475 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:44.232482 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:44.232513 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:44.232819 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:44.232832 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.233949 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.233954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:44.236273 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:44.236292 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.238223 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.238228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:44.238590 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:44.238756 (+ 166us) client_negotiation.cc:769] Sending connection context
0924 19:00:44.238876 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 19:00:44.239028 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":184}
I20260924 19:00:44.239925 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.231705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44205 (local address 127.0.97.3:44055)
0924 19:00:44.231887 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:44.231893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:44.231989 (+ 96us) server_negotiation.cc:407] Connection header received
0924 19:00:44.232265 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:44.232271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:44.232351 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:44.232444 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:44.232975 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.233816 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.236468 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.237870 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.237973 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:44.238064 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:44.239504 (+ 1440us) server_negotiation.cc:299] Negotiation successful
0924 19:00:44.239635 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 19:00:44.240329 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.231640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33575 (local address 127.0.97.2:45633)
0924 19:00:44.232322 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.232339 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:44.232464 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:44.232770 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:44.232773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:44.232797 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:44.233127 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:44.233137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.234290 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.234295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:44.236362 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:44.236376 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.238181 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.238189 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:44.238599 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:44.238756 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 19:00:44.239527 (+ 771us) client_negotiation.cc:240] Negotiation successful
0924 19:00:44.239641 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":157,"thread_start_us":407,"threads_started":1}
I20260924 19:00:44.240666 7761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "514b90182c4541b0842ac294e9193455" candidate_uuid: "56fd3560adc24fa5aa71678e700f9836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c576b1596f4264a2d7ce70c7ec7c4a" is_pre_election: true
I20260924 19:00:44.240935 7472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "514b90182c4541b0842ac294e9193455" candidate_uuid: "56fd3560adc24fa5aa71678e700f9836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b40f242c874e81a6ab82ce8cb210c7" is_pre_election: true
I20260924 19:00:44.241166 7472 raft_consensus.cc:2454] T 514b90182c4541b0842ac294e9193455 P 31b40f242c874e81a6ab82ce8cb210c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56fd3560adc24fa5aa71678e700f9836 in term 0.
I20260924 19:00:44.241130 7761 raft_consensus.cc:2454] T 514b90182c4541b0842ac294e9193455 P 83c576b1596f4264a2d7ce70c7ec7c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56fd3560adc24fa5aa71678e700f9836 in term 0.
I20260924 19:00:44.241694 7555 leader_election.cc:302] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 31b40f242c874e81a6ab82ce8cb210c7, 56fd3560adc24fa5aa71678e700f9836, 83c576b1596f4264a2d7ce70c7ec7c4a; no voters:
I20260924 19:00:44.242070 7905 raft_consensus.cc:2781] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:00:44.242208 7905 raft_consensus.cc:482] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:00:44.242286 7905 raft_consensus.cc:3037] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:44.245337 7905 raft_consensus.cc:504] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c576b1596f4264a2d7ce70c7ec7c4a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44055 } } peers { permanent_uuid: "56fd3560adc24fa5aa71678e700f9836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44407 } } peers { permanent_uuid: "31b40f242c874e81a6ab82ce8cb210c7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33575 } }
I20260924 19:00:44.246035 7905 leader_election.cc:288] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836 [CANDIDATE]: Term 1 election: Requested vote from peers 83c576b1596f4264a2d7ce70c7ec7c4a (127.0.97.3:44055), 31b40f242c874e81a6ab82ce8cb210c7 (127.0.97.1:33575)
I20260924 19:00:44.246505 7472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "514b90182c4541b0842ac294e9193455" candidate_uuid: "56fd3560adc24fa5aa71678e700f9836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31b40f242c874e81a6ab82ce8cb210c7"
I20260924 19:00:44.246516 7761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "514b90182c4541b0842ac294e9193455" candidate_uuid: "56fd3560adc24fa5aa71678e700f9836" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c576b1596f4264a2d7ce70c7ec7c4a"
I20260924 19:00:44.246711 7761 raft_consensus.cc:3037] T 514b90182c4541b0842ac294e9193455 P 83c576b1596f4264a2d7ce70c7ec7c4a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:44.246709 7472 raft_consensus.cc:3037] T 514b90182c4541b0842ac294e9193455 P 31b40f242c874e81a6ab82ce8cb210c7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:44.248811 7472 raft_consensus.cc:2454] T 514b90182c4541b0842ac294e9193455 P 31b40f242c874e81a6ab82ce8cb210c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56fd3560adc24fa5aa71678e700f9836 in term 1.
I20260924 19:00:44.249158 7555 leader_election.cc:302] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31b40f242c874e81a6ab82ce8cb210c7, 56fd3560adc24fa5aa71678e700f9836; no voters:
I20260924 19:00:44.249639 7905 raft_consensus.cc:2781] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:00:44.249795 7761 raft_consensus.cc:2454] T 514b90182c4541b0842ac294e9193455 P 83c576b1596f4264a2d7ce70c7ec7c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56fd3560adc24fa5aa71678e700f9836 in term 1.
I20260924 19:00:44.250476 7905 raft_consensus.cc:686] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836 [term 1 LEADER]: Becoming Leader. State: Replica: 56fd3560adc24fa5aa71678e700f9836, State: Running, Role: LEADER
I20260924 19:00:44.250921 7905 consensus_queue.cc:237] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c576b1596f4264a2d7ce70c7ec7c4a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44055 } } peers { permanent_uuid: "56fd3560adc24fa5aa71678e700f9836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44407 } } peers { permanent_uuid: "31b40f242c874e81a6ab82ce8cb210c7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33575 } }
I20260924 19:00:44.251537 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.251100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.97.2:55187)
0924 19:00:44.251283 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.251395 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 19:00:44.254266 7147 catalog_manager.cc:5495] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56fd3560adc24fa5aa71678e700f9836 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "56fd3560adc24fa5aa71678e700f9836" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c576b1596f4264a2d7ce70c7ec7c4a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56fd3560adc24fa5aa71678e700f9836" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31b40f242c874e81a6ab82ce8cb210c7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33575 } health_report { overall_health: UNKNOWN } } }
I20260924 19:00:44.273463 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.268507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34634 (local address 127.0.97.2:44407)
0924 19:00:44.268717 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:44.268723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:44.268878 (+ 155us) server_negotiation.cc:407] Connection header received
0924 19:00:44.269059 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:44.269062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:44.269148 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:44.269270 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:00:44.269612 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.270320 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.271560 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.272003 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.272093 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:44.272401 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:00:44.272926 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:00:44.273031 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:44.273098 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 19:00:44.273169 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 19:00:44.277333 7472 raft_consensus.cc:1261] T 514b90182c4541b0842ac294e9193455 P 31b40f242c874e81a6ab82ce8cb210c7 [term 1 FOLLOWER]: Refusing update from remote peer 56fd3560adc24fa5aa71678e700f9836: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:00:44.277696 7761 raft_consensus.cc:1261] T 514b90182c4541b0842ac294e9193455 P 83c576b1596f4264a2d7ce70c7ec7c4a [term 1 FOLLOWER]: Refusing update from remote peer 56fd3560adc24fa5aa71678e700f9836: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:00:44.278064 7912 consensus_queue.cc:1035] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31b40f242c874e81a6ab82ce8cb210c7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:00:44.278445 7905 consensus_queue.cc:1035] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c576b1596f4264a2d7ce70c7ec7c4a" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:00:44.281005 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.280432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.97.1:60219)
0924 19:00:44.280684 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.280781 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260924 19:00:44.285008 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.284419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.97.3:59333)
0924 19:00:44.284649 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.284770 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 19:00:44.287837 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.287204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.97.1:47595)
0924 19:00:44.287596 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.287693 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295}
I20260924 19:00:44.290370 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.287507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.97.2:46007)
0924 19:00:44.290032 (+ 2525us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.290161 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2374}
I20260924 19:00:44.298209 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.296044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.97.2:54707)
0924 19:00:44.297945 (+ 1901us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.298025 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1802}
I20260924 19:00:44.299415 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.298865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.97.3:54371)
0924 19:00:44.299108 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.299219 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260924 19:00:44.306077 7147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:00:44.306483 7147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:00:44.313596 7147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07bd526e5ab649bb873f289aee41b508: Sending DeleteTablet for 3 replicas of tablet 514b90182c4541b0842ac294e9193455
I20260924 19:00:44.314404 7741 tablet_service.cc:1514] Processing DeleteTablet for tablet 514b90182c4541b0842ac294e9193455 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:44 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:51742
I20260924 19:00:44.314980 7452 tablet_service.cc:1514] Processing DeleteTablet for tablet 514b90182c4541b0842ac294e9193455 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:44 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:40224
I20260924 19:00:44.315048 7925 tablet_replica.cc:330] T 514b90182c4541b0842ac294e9193455 P 83c576b1596f4264a2d7ce70c7ec7c4a: stopping tablet replica
I20260924 19:00:44.315297 7925 raft_consensus.cc:2229] T 514b90182c4541b0842ac294e9193455 P 83c576b1596f4264a2d7ce70c7ec7c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:00:44.315587 7926 tablet_replica.cc:330] T 514b90182c4541b0842ac294e9193455 P 31b40f242c874e81a6ab82ce8cb210c7: stopping tablet replica
I20260924 19:00:44.315605 7600 tablet_service.cc:1514] Processing DeleteTablet for tablet 514b90182c4541b0842ac294e9193455 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:44 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:34632
I20260924 19:00:44.315754 7926 raft_consensus.cc:2229] T 514b90182c4541b0842ac294e9193455 P 31b40f242c874e81a6ab82ce8cb210c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:00:44.317421 7741 tablet_service.cc:1514] Processing DeleteTablet for tablet 514b90182c4541b0842ac294e9193455 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:44 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:51742
I20260924 19:00:44.317588 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 7325
I20260924 19:00:44.316020 7926 raft_consensus.cc:2258] T 514b90182c4541b0842ac294e9193455 P 31b40f242c874e81a6ab82ce8cb210c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:44.321267 7926 ts_tablet_manager.cc:1905] T 514b90182c4541b0842ac294e9193455 P 31b40f242c874e81a6ab82ce8cb210c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:44.321818 7925 raft_consensus.cc:2258] T 514b90182c4541b0842ac294e9193455 P 83c576b1596f4264a2d7ce70c7ec7c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:44.323267 7925 ts_tablet_manager.cc:1905] T 514b90182c4541b0842ac294e9193455 P 83c576b1596f4264a2d7ce70c7ec7c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:44.327080 7926 ts_tablet_manager.cc:1918] T 514b90182c4541b0842ac294e9193455 P 31b40f242c874e81a6ab82ce8cb210c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:00:44.327220 7926 log.cc:1196] T 514b90182c4541b0842ac294e9193455 P 31b40f242c874e81a6ab82ce8cb210c7: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-0/wal/wals/514b90182c4541b0842ac294e9193455
I20260924 19:00:44.327752 7926 ts_tablet_manager.cc:1939] T 514b90182c4541b0842ac294e9193455 P 31b40f242c874e81a6ab82ce8cb210c7: Deleting consensus metadata
I20260924 19:00:44.329013 7134 catalog_manager.cc:4841] TS 31b40f242c874e81a6ab82ce8cb210c7 (127.0.97.1:33575): tablet 514b90182c4541b0842ac294e9193455 (table test-table [id=0939c3b6892d46cba8c963b13c56163b]) successfully deleted
I20260924 19:00:44.330020 7927 tablet_replica.cc:330] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836: stopping tablet replica
I20260924 19:00:44.330228 7927 raft_consensus.cc:2229] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:00:44.330834 7927 raft_consensus.cc:2258] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:44.332289 7925 ts_tablet_manager.cc:1918] T 514b90182c4541b0842ac294e9193455 P 83c576b1596f4264a2d7ce70c7ec7c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:00:44.332391 7925 log.cc:1196] T 514b90182c4541b0842ac294e9193455 P 83c576b1596f4264a2d7ce70c7ec7c4a: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-2/wal/wals/514b90182c4541b0842ac294e9193455
I20260924 19:00:44.332801 7925 ts_tablet_manager.cc:1939] T 514b90182c4541b0842ac294e9193455 P 83c576b1596f4264a2d7ce70c7ec7c4a: Deleting consensus metadata
I20260924 19:00:44.333777 7132 catalog_manager.cc:4841] TS 83c576b1596f4264a2d7ce70c7ec7c4a (127.0.97.3:44055): tablet 514b90182c4541b0842ac294e9193455 (table test-table [id=0939c3b6892d46cba8c963b13c56163b]) successfully deleted
W20260924 19:00:44.334215 7132 catalog_manager.cc:4808] TS 83c576b1596f4264a2d7ce70c7ec7c4a (127.0.97.3:44055): delete failed for tablet 514b90182c4541b0842ac294e9193455 because the tablet was not found. No further retry: Not found: Tablet not found: 514b90182c4541b0842ac294e9193455
I20260924 19:00:44.335575 7927 ts_tablet_manager.cc:1905] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:44.337917 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.318581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40228 (local address 127.0.97.1:33575)
0924 19:00:44.318817 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:44.318827 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:44.320830 (+ 2003us) server_negotiation.cc:407] Connection header received
0924 19:00:44.320916 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:44.320923 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:44.321025 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:44.321153 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:44.322800 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.323966 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.325958 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.326570 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.326752 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:44.330742 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:44.330759 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:44.330782 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:44.330826 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:44.334808 (+ 3982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:44.335647 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:44.335655 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:44.335658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:44.335778 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:44.336155 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:44.336160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:44.336168 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:44.336556 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:44.336903 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:44.337255 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 19:00:44.337489 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 19:00:44.338511 7512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:44.344256 7927 ts_tablet_manager.cc:1918] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:00:44.344431 7927 log.cc:1196] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276386554690-388-0/minicluster-data/ts-1/wal/wals/514b90182c4541b0842ac294e9193455
I20260924 19:00:44.344959 7927 ts_tablet_manager.cc:1939] T 514b90182c4541b0842ac294e9193455 P 56fd3560adc24fa5aa71678e700f9836: Deleting consensus metadata
I20260924 19:00:44.346336 7135 catalog_manager.cc:4841] TS 56fd3560adc24fa5aa71678e700f9836 (127.0.97.2:44407): tablet 514b90182c4541b0842ac294e9193455 (table test-table [id=0939c3b6892d46cba8c963b13c56163b]) successfully deleted
I20260924 19:00:44.431874 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 7325
I20260924 19:00:44.449692 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 7529
I20260924 19:00:44.464769 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.450502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33258 (local address 127.0.97.2:44407)
0924 19:00:44.450683 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:44.450690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:44.450718 (+ 28us) server_negotiation.cc:407] Connection header received
0924 19:00:44.450897 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:44.450902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:44.450981 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:44.451097 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:44.452283 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.453478 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.454743 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.455402 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.455528 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:44.458113 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:44.458138 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:44.458158 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:44.458196 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:44.461965 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:44.462563 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:44.462572 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:44.462576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:44.462769 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:44.463158 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:44.463163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:44.463167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:44.463642 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:44.463947 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:44.464174 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 19:00:44.464382 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:00:44.465441 7660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:44.582130 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 7529
I20260924 19:00:44.602083 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 7678
I20260924 19:00:44.614053 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.602814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52300 (local address 127.0.97.3:44055)
0924 19:00:44.602959 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:44.602962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:44.602992 (+ 30us) server_negotiation.cc:407] Connection header received
0924 19:00:44.603146 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:44.603149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:44.603208 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:44.603290 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:44.604506 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.605198 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.606451 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:44.606812 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:44.606906 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:44.609210 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:44.609223 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:44.609239 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:44.609279 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:44.611870 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:44.612433 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:44.612439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:44.612441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:44.612554 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:44.612847 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:44.612850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:44.612852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:44.613146 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:44.613381 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:44.613604 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 19:00:44.613759 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 19:00:44.614739 7801 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:44.650509 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.650124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.0.1:38736)
0924 19:00:44.650297 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.650375 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 19:00:44.651820 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:44.651483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46537 (local address 127.0.0.1:38740)
0924 19:00:44.651638 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:44.651717 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 19:00:44.730201 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 7678
I20260924 19:00:44.751488 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 7011
I20260924 19:00:44.752552 7178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:44.871384 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 7011
W20260924 19:00:44.892056 7323 connection.cc:535] server connection from 127.0.0.1:46312 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 19:00:44.892367 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 7107
I20260924 19:00:44.893357 7368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:45.006222 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 7107
2026-09-24T19:00:45Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9332 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:00:45 dist-test-slave-9fn1 krb5kdc[7937](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:00:45 dist-test-slave-9fn1 krb5kdc[7937](info): set up 2 sockets
Sep 24 19:00:45 dist-test-slave-9fn1 krb5kdc[7937](info): commencing operation
krb5kdc: starting...
W20260924 19:00:46.090355 388 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:00:46 dist-test-slave-9fn1 krb5kdc[7937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:00:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:00:46Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260924 19:00:46.292479 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38989
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:46.606181 7950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:46.606439 7950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:46.606611 7950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:46.613723 7950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:00:46.613812 7950 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 19:00:46.613876 7950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:46.613993 7950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:46.614101 7950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:00:46.614233 7950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:00:46.623631 7950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38989
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:46.624857 7950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:46.626161 7950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:46.637874 7955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:46.637923 7956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:46.637972 7958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:46.638777 7950 server_base.cc:1034] running on GCE node
I20260924 19:00:46.639531 7950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:46.640815 7950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:46.642047 7950 hybrid_clock.cc:648] HybridClock initialized: now 1790276446641990 us; error 34 us; skew 500 ppm
I20260924 19:00:46.642359 7950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:46 dist-test-slave-9fn1 krb5kdc[7937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276446, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:46.645308 7950 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260924 19:00:46.646768 7950 webserver.cc:460] Webserver started at http://127.0.97.62:45561/ using document root <none> and password file <none>
I20260924 19:00:46.647460 7950 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:46.647588 7950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:46.647895 7950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:46.650837 7950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "5a099787801042bcb43901bb50891277"
format_stamp: "Formatted at 2026-09-24 19:00:46 on dist-test-slave-9fn1"
server_key: "496529fa427dd7b0c4827c6f1258ee46"
server_key_iv: "0062dfb406a00295f77656b1ac8a449b"
server_key_version: "encryptionkey@0"
I20260924 19:00:46.651644 7950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "5a099787801042bcb43901bb50891277"
format_stamp: "Formatted at 2026-09-24 19:00:46 on dist-test-slave-9fn1"
server_key: "496529fa427dd7b0c4827c6f1258ee46"
server_key_iv: "0062dfb406a00295f77656b1ac8a449b"
server_key_version: "encryptionkey@0"
I20260924 19:00:46.656702 7950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260924 19:00:46.660050 7965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:46.661283 7950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 19:00:46.661460 7950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "5a099787801042bcb43901bb50891277"
format_stamp: "Formatted at 2026-09-24 19:00:46 on dist-test-slave-9fn1"
server_key: "496529fa427dd7b0c4827c6f1258ee46"
server_key_iv: "0062dfb406a00295f77656b1ac8a449b"
server_key_version: "encryptionkey@0"
I20260924 19:00:46.661734 7950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:46.681068 7950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:46.684983 7950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:46.685261 7950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:46.707407 7950 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38989
I20260924 19:00:46.707427 8026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38989 every 8 connection(s)
I20260924 19:00:46.708786 7950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:00:46.711153 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 7950
I20260924 19:00:46.711369 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:00:46.712097 388 external_mini_cluster.cc:1428] Setting key 634f03d06857fd9aeea856453872c46c
I20260924 19:00:46.713048 8027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:46.727284 8027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a099787801042bcb43901bb50891277: Bootstrap starting.
Sep 24 19:00:46 dist-test-slave-9fn1 krb5kdc[7937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 19:00:46.731014 8027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a099787801042bcb43901bb50891277: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:46.732100 8027 log.cc:824] T 00000000000000000000000000000000 P 5a099787801042bcb43901bb50891277: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:46.735180 8027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a099787801042bcb43901bb50891277: No bootstrap required, opened a new log
I20260924 19:00:46.735859 8030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:46.714105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53314 (local address 127.0.97.62:38989)
0924 19:00:46.714909 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:46.714927 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:46.714984 (+ 57us) server_negotiation.cc:407] Connection header received
0924 19:00:46.716770 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:46.716797 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:46.717301 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:46.717732 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:46.719599 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:46.721633 (+ 2034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:46.722860 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:46.723862 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:46.724051 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:46.728777 (+ 4726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:46.728785 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:46.728798 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:46.728838 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:46.731804 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:46.732594 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:46.732608 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:46.732631 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:46.732848 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:46.733182 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:46.733188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:46.733191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:46.733804 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:46.733991 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:46.734450 (+ 459us) server_negotiation.cc:299] Negotiation successful
0924 19:00:46.734781 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":369,"threads_started":1}
I20260924 19:00:46.743563 8027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a099787801042bcb43901bb50891277 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a099787801042bcb43901bb50891277" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38989 } }
I20260924 19:00:46.744045 8027 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a099787801042bcb43901bb50891277 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:46.744148 8027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a099787801042bcb43901bb50891277 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a099787801042bcb43901bb50891277, State: Initialized, Role: FOLLOWER
I20260924 19:00:46.744822 8027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a099787801042bcb43901bb50891277 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a099787801042bcb43901bb50891277" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38989 } }
I20260924 19:00:46.745122 8027 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a099787801042bcb43901bb50891277 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:00:46.745257 8027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a099787801042bcb43901bb50891277 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:00:46.745450 8027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a099787801042bcb43901bb50891277 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:46.748498 8027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a099787801042bcb43901bb50891277 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a099787801042bcb43901bb50891277" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38989 } }
I20260924 19:00:46.749007 8027 leader_election.cc:302] T 00000000000000000000000000000000 P 5a099787801042bcb43901bb50891277 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a099787801042bcb43901bb50891277; no voters:
I20260924 19:00:46.749686 8027 leader_election.cc:288] T 00000000000000000000000000000000 P 5a099787801042bcb43901bb50891277 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:00:46.749838 8032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a099787801042bcb43901bb50891277 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:00:46.750613 8032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a099787801042bcb43901bb50891277 [term 1 LEADER]: Becoming Leader. State: Replica: 5a099787801042bcb43901bb50891277, State: Running, Role: LEADER
I20260924 19:00:46.751194 8032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a099787801042bcb43901bb50891277 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a099787801042bcb43901bb50891277" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38989 } }
I20260924 19:00:46.753312 8032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a099787801042bcb43901bb50891277 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a099787801042bcb43901bb50891277. Latest consensus state: current_term: 1 leader_uuid: "5a099787801042bcb43901bb50891277" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a099787801042bcb43901bb50891277" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38989 } } }
I20260924 19:00:46.753598 8032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a099787801042bcb43901bb50891277 [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:46.754138 8034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a099787801042bcb43901bb50891277 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a099787801042bcb43901bb50891277" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a099787801042bcb43901bb50891277" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38989 } } }
I20260924 19:00:46.754441 8034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a099787801042bcb43901bb50891277 [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:46.755415 8027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a099787801042bcb43901bb50891277 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:00:46.755568 8036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:00:46.762310 8036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:00:46.770607 8036 catalog_manager.cc:1347] Generated new cluster ID: 5729ed3a9f7b4baebfe046cf4ef71bcd
I20260924 19:00:46.770730 8036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:00:46.780671 8036 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:00:46.782480 8036 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:00:46.796933 8036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a099787801042bcb43901bb50891277: Generated new TSK 0
I20260924 19:00:46.797996 8036 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260924 19:00:46.845561 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38989
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:47.167990 8053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:47.168264 8053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:47.168358 8053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:47.175436 8053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:47.175535 8053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:47.175628 8053 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:00:47.175706 8053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:00:47.185065 8053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:47.186244 8053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:47.187476 8053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:47.200155 8058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:47.200183 8059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:47.200227 8061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:47.200991 8053 server_base.cc:1034] running on GCE node
I20260924 19:00:47.201555 8053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:47.202394 8053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:47.203622 8053 hybrid_clock.cc:648] HybridClock initialized: now 1790276447203548 us; error 31 us; skew 500 ppm
I20260924 19:00:47.203950 8053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:47 dist-test-slave-9fn1 krb5kdc[7937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276447, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:47.207000 8053 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260924 19:00:47.208393 8053 webserver.cc:460] Webserver started at http://127.0.97.1:41157/ using document root <none> and password file <none>
I20260924 19:00:47.209062 8053 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:47.209170 8053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:47.209486 8053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:47.212380 8053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-0/data/instance:
uuid: "934856ed75d54e2cbc38989df8f509d1"
format_stamp: "Formatted at 2026-09-24 19:00:47 on dist-test-slave-9fn1"
server_key: "b231d482f6179b4db1b744c40b146c73"
server_key_iv: "55942ab57d01cadc01489573546c82a8"
server_key_version: "encryptionkey@0"
I20260924 19:00:47.213164 8053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-0/wal/instance:
uuid: "934856ed75d54e2cbc38989df8f509d1"
format_stamp: "Formatted at 2026-09-24 19:00:47 on dist-test-slave-9fn1"
server_key: "b231d482f6179b4db1b744c40b146c73"
server_key_iv: "55942ab57d01cadc01489573546c82a8"
server_key_version: "encryptionkey@0"
I20260924 19:00:47.218041 8053 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.006s
I20260924 19:00:47.221227 8068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:47.222523 8053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:00:47.222687 8053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "934856ed75d54e2cbc38989df8f509d1"
format_stamp: "Formatted at 2026-09-24 19:00:47 on dist-test-slave-9fn1"
server_key: "b231d482f6179b4db1b744c40b146c73"
server_key_iv: "55942ab57d01cadc01489573546c82a8"
server_key_version: "encryptionkey@0"
I20260924 19:00:47.222892 8053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:47.234319 8053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:47.238425 8053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:47.238680 8053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:47.241199 8053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:47.241318 8053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:47.241461 8053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:47.241555 8053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 24 19:00:47 dist-test-slave-9fn1 krb5kdc[7937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276447, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 19:00:47.265475 8076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:47.241591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59353 (local address 127.0.97.62:38989)
0924 19:00:47.242101 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:47.242110 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:47.243196 (+ 1086us) server_negotiation.cc:407] Connection header received
0924 19:00:47.245122 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:47.245128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:47.245218 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:47.245338 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:47.247609 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:47.248703 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:47.251858 (+ 3155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:47.252409 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:47.252591 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:47.258443 (+ 5852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:47.258451 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:47.258455 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:47.258499 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:47.261447 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:47.262534 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:47.262539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:47.262541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:47.262658 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:47.263237 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:47.263241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:47.263243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:47.263484 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:47.263596 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:47.264619 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0924 19:00:47.264948 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":264,"threads_started":1}
I20260924 19:00:47.267249 8078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:47.242246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38989 (local address 127.0.97.1:59353)
0924 19:00:47.243027 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:47.243075 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:47.244852 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:47.245627 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:47.245635 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:47.246405 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:47.247358 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:47.247389 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:47.248901 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:47.248910 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:47.251718 (+ 2808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:47.251733 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:47.252923 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:47.252930 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:47.253210 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:47.254127 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:47.254164 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:47.258154 (+ 3990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:47.261638 (+ 3484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:47.261651 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:47.261672 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:47.262391 (+ 719us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:47.262768 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:47.262775 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:47.262782 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:47.263110 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:47.263618 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:47.263628 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:47.264144 (+ 516us) client_negotiation.cc:769] Sending connection context
0924 19:00:47.264534 (+ 390us) client_negotiation.cc:240] Negotiation successful
0924 19:00:47.265267 (+ 733us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":314,"threads_started":1}
I20260924 19:00:47.286535 8053 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40683
I20260924 19:00:47.286607 8188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40683 every 8 connection(s)
I20260924 19:00:47.287707 8053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:00:47.289834 8189 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38989
I20260924 19:00:47.290066 8189 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:47.290807 8189 heartbeater.cc:511] Master 127.0.97.62:38989 requested a full tablet report, sending...
I20260924 19:00:47.292908 7982 ts_manager.cc:194] Registered new tserver with Master: 934856ed75d54e2cbc38989df8f509d1 (127.0.97.1:40683)
I20260924 19:00:47.294456 7982 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59353
I20260924 19:00:47.295517 8076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:47.280526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51401 (local address 127.0.97.62:38989)
0924 19:00:47.280707 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:47.280714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:47.281898 (+ 1184us) server_negotiation.cc:407] Connection header received
0924 19:00:47.282192 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:47.282199 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:47.282282 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:47.282357 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:47.284045 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:47.285144 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:47.287242 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:47.287652 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:47.287795 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:47.290105 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:47.290112 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:47.290116 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:47.290156 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:47.292922 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:47.293581 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:47.293586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:47.293589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:47.293724 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:47.294108 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:47.294113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:47.294116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:47.294372 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:47.294548 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:47.294950 (+ 402us) server_negotiation.cc:299] Negotiation successful
0924 19:00:47.295197 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 19:00:47.295522 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 8053
I20260924 19:00:47.295607 8172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:47.280752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38989 (local address 127.0.97.1:51401)
0924 19:00:47.281777 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:47.281803 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:47.282022 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:47.282600 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:47.282607 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:47.283269 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:47.283853 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:47.283871 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:47.285361 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:47.285369 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:47.287099 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:47.287111 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:47.287882 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:47.287888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:47.288038 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:47.288766 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:47.288792 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:47.289899 (+ 1107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:47.293058 (+ 3159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:47.293065 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:47.293071 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:47.293422 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:47.293853 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:47.293860 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:47.293865 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:47.293991 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:47.294493 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:47.294498 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:47.294683 (+ 185us) client_negotiation.cc:769] Sending connection context
0924 19:00:47.294848 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 19:00:47.295139 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":489,"threads_started":1}
I20260924 19:00:47.295712 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-0/wal/instance
I20260924 19:00:47.296512 388 external_mini_cluster.cc:1428] Setting key 981bfea8dc3db1679b9d6eee213e4659
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260924 19:00:47.341981 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38989
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:47.668932 8193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:47.669231 8193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:47.669400 8193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:47.676483 8193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:47.676623 8193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:47.676790 8193 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:00:47.676939 8193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:00:47.686424 8193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:47.687683 8193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:47.688988 8193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:47.702239 8198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:47.702239 8199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:47.702267 8201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:47.703039 8193 server_base.cc:1034] running on GCE node
I20260924 19:00:47.703563 8193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:47.704366 8193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:47.705626 8193 hybrid_clock.cc:648] HybridClock initialized: now 1790276447705548 us; error 42 us; skew 500 ppm
I20260924 19:00:47.705940 8193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:47 dist-test-slave-9fn1 krb5kdc[7937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276447, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:47.709492 8193 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260924 19:00:47.714561 8193 webserver.cc:460] Webserver started at http://127.0.97.2:44921/ using document root <none> and password file <none>
I20260924 19:00:47.715224 8193 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:47.715348 8193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:47.715669 8193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:47.718459 8193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-1/data/instance:
uuid: "69ea0ce743ff4c8ca80afa6c7da26941"
format_stamp: "Formatted at 2026-09-24 19:00:47 on dist-test-slave-9fn1"
server_key: "c4505d83ad6c65720cae4aa1d0cee3df"
server_key_iv: "d0452224410fcab9af67a1627be0a19d"
server_key_version: "encryptionkey@0"
I20260924 19:00:47.719231 8193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-1/wal/instance:
uuid: "69ea0ce743ff4c8ca80afa6c7da26941"
format_stamp: "Formatted at 2026-09-24 19:00:47 on dist-test-slave-9fn1"
server_key: "c4505d83ad6c65720cae4aa1d0cee3df"
server_key_iv: "d0452224410fcab9af67a1627be0a19d"
server_key_version: "encryptionkey@0"
I20260924 19:00:47.724277 8193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:00:47.727653 8208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:47.728852 8193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 19:00:47.729024 8193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "69ea0ce743ff4c8ca80afa6c7da26941"
format_stamp: "Formatted at 2026-09-24 19:00:47 on dist-test-slave-9fn1"
server_key: "c4505d83ad6c65720cae4aa1d0cee3df"
server_key_iv: "d0452224410fcab9af67a1627be0a19d"
server_key_version: "encryptionkey@0"
I20260924 19:00:47.729233 8193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:47.770557 8193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:47.775157 8193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:47.775468 8193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:47.778440 8193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:47.778558 8193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:47.778709 8193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:47.778792 8193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:00:47 dist-test-slave-9fn1 krb5kdc[7937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276447, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 19:00:47.805975 8217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:47.779678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38989 (local address 127.0.97.2:41087)
0924 19:00:47.780553 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:47.780590 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:47.782548 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:47.783304 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:47.783313 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:47.784064 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:47.785135 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:47.785170 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:47.786571 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:47.786577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:47.788882 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:47.788901 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:47.790086 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:47.790094 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:47.790342 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:47.791138 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:47.791170 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:47.795368 (+ 4198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:47.800451 (+ 5083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:47.800467 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:47.800487 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:47.801251 (+ 764us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:47.801809 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:47.801825 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:47.801830 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:47.802115 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:47.802815 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:47.802833 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:47.803334 (+ 501us) client_negotiation.cc:769] Sending connection context
0924 19:00:47.803678 (+ 344us) client_negotiation.cc:240] Negotiation successful
0924 19:00:47.804276 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":407,"threads_started":1}
I20260924 19:00:47.806847 8076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:47.778936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41087 (local address 127.0.97.62:38989)
0924 19:00:47.779138 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:47.779146 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:47.780720 (+ 1574us) server_negotiation.cc:407] Connection header received
0924 19:00:47.782855 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:47.782860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:47.782955 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:47.783062 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:47.785326 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:47.786427 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:47.789062 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:47.789792 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:47.789949 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:47.795725 (+ 5776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:47.795732 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:47.795737 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:47.795807 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:47.800251 (+ 4444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:47.801429 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:47.801439 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:47.801445 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:47.801647 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:47.802242 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:47.802249 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:47.802254 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:47.802660 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:47.802819 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:47.806467 (+ 3648us) server_negotiation.cc:299] Negotiation successful
0924 19:00:47.806621 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 19:00:47.825214 8076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:47.811720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49773 (local address 127.0.97.62:38989)
0924 19:00:47.811928 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:47.811936 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:47.812591 (+ 655us) server_negotiation.cc:407] Connection header received
0924 19:00:47.812775 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:47.812778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:47.812834 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:47.812908 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:47.814406 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:47.815108 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:47.817154 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:47.817744 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:47.817875 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:47.819561 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:47.819567 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:47.819571 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:47.819609 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:47.822430 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:47.823303 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:47.823308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:47.823322 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:47.823466 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:47.823930 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:47.823933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:47.823935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:47.824219 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:47.824330 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:47.824854 (+ 524us) server_negotiation.cc:299] Negotiation successful
0924 19:00:47.825008 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 19:00:47.826001 8302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:47.811720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38989 (local address 127.0.97.2:49773)
0924 19:00:47.812475 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:47.812497 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:47.812663 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:47.812973 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:47.812981 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:47.813722 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:47.814237 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:47.814256 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:47.815262 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:47.815266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:47.816971 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:47.816990 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:47.817943 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:47.817949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:47.818096 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:47.818709 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:47.818726 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:47.819395 (+ 669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:47.822557 (+ 3162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:47.822565 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:47.822569 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:47.823182 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:47.823597 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:47.823605 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:47.823612 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:47.823806 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:47.824334 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:47.824342 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:47.824600 (+ 258us) client_negotiation.cc:769] Sending connection context
0924 19:00:47.824780 (+ 180us) client_negotiation.cc:240] Negotiation successful
0924 19:00:47.825218 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":401,"threads_started":1}
I20260924 19:00:47.832018 8193 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46793
I20260924 19:00:47.832123 8327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46793 every 8 connection(s)
I20260924 19:00:47.833240 8193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:00:47.835206 8328 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38989
I20260924 19:00:47.835381 8328 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:47.835944 8328 heartbeater.cc:511] Master 127.0.97.62:38989 requested a full tablet report, sending...
I20260924 19:00:47.837110 7981 ts_manager.cc:194] Registered new tserver with Master: 69ea0ce743ff4c8ca80afa6c7da26941 (127.0.97.2:46793)
I20260924 19:00:47.837986 7981 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41087
I20260924 19:00:47.842193 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 8193
I20260924 19:00:47.842363 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-1/wal/instance
I20260924 19:00:47.842780 388 external_mini_cluster.cc:1428] Setting key ee7a77a987464f582684608bfae4c9f5
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260924 19:00:47.888367 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38989
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:48.203408 8332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:48.203707 8332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:48.203876 8332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:48.211144 8332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:48.211294 8332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:48.211450 8332 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:00:48.211582 8332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:00:48.221310 8332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:38989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:48.222537 8332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:48.223824 8332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:48.238157 8338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:48.238096 8337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:48.238389 8340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:48.239689 8332 server_base.cc:1034] running on GCE node
I20260924 19:00:48.240283 8332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:48.241142 8332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:48.242408 8332 hybrid_clock.cc:648] HybridClock initialized: now 1790276448242332 us; error 44 us; skew 500 ppm
I20260924 19:00:48.242738 8332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:48 dist-test-slave-9fn1 krb5kdc[7937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276448, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:48.245839 8332 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260924 19:00:48.247203 8332 webserver.cc:460] Webserver started at http://127.0.97.3:40861/ using document root <none> and password file <none>
I20260924 19:00:48.247933 8332 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:48.248061 8332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:48.248379 8332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:48.251680 8332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-2/data/instance:
uuid: "e31a53118f4d45ad946d20cf4952a46b"
format_stamp: "Formatted at 2026-09-24 19:00:48 on dist-test-slave-9fn1"
server_key: "c87d94742a9c18af0a8a6ebae491d359"
server_key_iv: "c171aafc2b091d04ec25cc5059da2271"
server_key_version: "encryptionkey@0"
I20260924 19:00:48.252481 8332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e31a53118f4d45ad946d20cf4952a46b"
format_stamp: "Formatted at 2026-09-24 19:00:48 on dist-test-slave-9fn1"
server_key: "c87d94742a9c18af0a8a6ebae491d359"
server_key_iv: "c171aafc2b091d04ec25cc5059da2271"
server_key_version: "encryptionkey@0"
I20260924 19:00:48.257716 8332 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 19:00:48.261083 8347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:48.262404 8332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:00:48.262573 8332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "e31a53118f4d45ad946d20cf4952a46b"
format_stamp: "Formatted at 2026-09-24 19:00:48 on dist-test-slave-9fn1"
server_key: "c87d94742a9c18af0a8a6ebae491d359"
server_key_iv: "c171aafc2b091d04ec25cc5059da2271"
server_key_version: "encryptionkey@0"
I20260924 19:00:48.262794 8332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:48.298302 8189 heartbeater.cc:503] Master 127.0.97.62:38989 was elected leader, sending a full tablet report...
I20260924 19:00:48.299392 8332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:48.303419 8332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:48.303705 8332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:48.306351 8332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:48.306478 8332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:48.306632 8332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:48.306705 8332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:00:48 dist-test-slave-9fn1 krb5kdc[7937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276448, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 19:00:48.330178 8076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:48.306825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39635 (local address 127.0.97.62:38989)
0924 19:00:48.307001 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:48.307010 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:48.308447 (+ 1437us) server_negotiation.cc:407] Connection header received
0924 19:00:48.310620 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:48.310625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:48.310715 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:48.310824 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:48.313040 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.314373 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.316494 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.317223 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.317388 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:48.322593 (+ 5205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:48.322601 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:48.322607 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:48.322651 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:48.326148 (+ 3497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:48.327116 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:48.327121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:48.327125 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:48.327323 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:48.327851 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:48.327860 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:48.327865 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:48.328358 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:48.328585 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:48.329354 (+ 769us) server_negotiation.cc:299] Negotiation successful
0924 19:00:48.329692 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 19:00:48.331014 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:48.307458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38989 (local address 127.0.97.3:39635)
0924 19:00:48.308300 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:48.308341 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:48.310351 (+ 2010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:48.311092 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:48.311106 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:48.311922 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:48.312761 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:48.312782 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.314571 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.314579 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:48.316348 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:48.316359 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.317547 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.317555 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:48.317763 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:48.318577 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:48.318619 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:48.322251 (+ 3632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:48.326338 (+ 4087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:48.326348 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:48.326377 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:48.326982 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:48.327439 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:48.327446 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:48.327450 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:48.327729 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:48.328544 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:48.328553 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:48.328921 (+ 368us) client_negotiation.cc:769] Sending connection context
0924 19:00:48.329166 (+ 245us) client_negotiation.cc:240] Negotiation successful
0924 19:00:48.329663 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":383,"threads_started":1}
I20260924 19:00:48.356518 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:48.340850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38989 (local address 127.0.97.3:52207)
0924 19:00:48.341615 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:48.341639 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:48.341846 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:48.342465 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:48.342472 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:48.343418 (+ 946us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:48.343965 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:48.343985 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.345603 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.345611 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:48.346621 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:48.346631 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.347820 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.347825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:48.347928 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:48.348565 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:48.348582 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:48.349352 (+ 770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:48.353390 (+ 4038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:48.353395 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:48.353403 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:48.354123 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:48.354601 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:48.354605 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:48.354609 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:48.354797 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:48.355548 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:48.355556 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:48.355718 (+ 162us) client_negotiation.cc:769] Sending connection context
0924 19:00:48.355846 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 19:00:48.356073 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":434,"threads_started":1}
I20260924 19:00:48.356863 8076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:48.340800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52207 (local address 127.0.97.62:38989)
0924 19:00:48.341018 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:48.341027 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:48.341734 (+ 707us) server_negotiation.cc:407] Connection header received
0924 19:00:48.342024 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:48.342030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:48.342133 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:48.342267 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:48.344184 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.345434 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.346760 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.347640 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.347796 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:48.349575 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:48.349583 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:48.349591 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:48.349635 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:48.353243 (+ 3608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:48.354314 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:48.354321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:48.354326 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:48.354489 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:48.354934 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:48.354941 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:48.354945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:48.355395 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:48.355582 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:48.356014 (+ 432us) server_negotiation.cc:299] Negotiation successful
0924 19:00:48.356352 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 19:00:48.364447 8332 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35721
I20260924 19:00:48.364528 8466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35721 every 8 connection(s)
I20260924 19:00:48.365824 8332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:00:48.368253 8467 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38989
I20260924 19:00:48.368438 8467 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:48.369117 8467 heartbeater.cc:511] Master 127.0.97.62:38989 requested a full tablet report, sending...
I20260924 19:00:48.369477 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 8332
I20260924 19:00:48.369673 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-2/wal/instance
I20260924 19:00:48.370234 388 external_mini_cluster.cc:1428] Setting key e257be5e00b6328520a04490cebbf973
I20260924 19:00:48.370618 7981 ts_manager.cc:194] Registered new tserver with Master: e31a53118f4d45ad946d20cf4952a46b (127.0.97.3:35721)
I20260924 19:00:48.371704 7981 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39635
I20260924 19:00:48.373004 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 19:00:48 dist-test-slave-9fn1 krb5kdc[7937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276448, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 19:00:48 dist-test-slave-9fn1 krb5kdc[7937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:00:48 dist-test-slave-9fn1 krb5kdc[7937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 19:00:48.419483 8076 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:00:48.409073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53318 (local address 127.0.97.62:38989)
0924 19:00:48.409346 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:48.409353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:48.409491 (+ 138us) server_negotiation.cc:407] Connection header received
0924 19:00:48.409739 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:48.409744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:48.409824 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:48.409929 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:48.411172 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.411876 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.413166 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.413655 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.413784 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:48.419145 (+ 5361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53318: BlockingRecv error: recv got EOF from 127.0.0.1:53318 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 24 19:00:48 dist-test-slave-9fn1 krb5kdc[7937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276448, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 19:00:48.434828 8076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:48.422339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53330 (local address 127.0.97.62:38989)
0924 19:00:48.422560 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:48.422566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:48.422779 (+ 213us) server_negotiation.cc:407] Connection header received
0924 19:00:48.422878 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:48.422881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:48.422951 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:48.423031 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:48.424043 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.424726 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.425948 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.426404 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.426544 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:48.430007 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:48.430012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:48.430015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:48.430043 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:48.432550 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:48.433269 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:48.433274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:48.433283 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:48.433402 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:48.433786 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:48.433789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:48.433791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:48.434049 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:48.434239 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:48.434444 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 19:00:48.434590 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 19:00:48.438681 7981 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:00:48.445212 7981 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:00:48.467943 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:48.456630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48440 (local address 127.0.97.1:40683)
0924 19:00:48.457375 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:48.457386 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:48.457987 (+ 601us) server_negotiation.cc:407] Connection header received
0924 19:00:48.458321 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:48.458330 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:48.458538 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:48.458807 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:48.459529 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.461035 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.464297 (+ 3262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.466258 (+ 1961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.466389 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:48.466661 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:48.467463 (+ 802us) server_negotiation.cc:299] Negotiation successful
0924 19:00:48.467633 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":449,"threads_started":1}
I20260924 19:00:48.468779 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:48.456540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40683 (local address 127.0.0.1:48440)
0924 19:00:48.457834 (+ 1294us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:48.457887 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:48.458052 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:48.458810 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:48.458817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:48.458844 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:48.459303 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:48.459317 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.461264 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.461272 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:48.464077 (+ 2805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:48.464102 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.466892 (+ 2790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.466901 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:48.467174 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:48.467229 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 19:00:48.468200 (+ 971us) client_negotiation.cc:240] Negotiation successful
0924 19:00:48.468412 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1058,"mutex_wait_us":10,"thread_start_us":909,"threads_started":1}
I20260924 19:00:48.470927 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:48.458233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35721 (local address 127.0.0.1:52416)
0924 19:00:48.459792 (+ 1559us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:48.459816 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:48.459925 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:48.460972 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:48.460980 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:48.461011 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:48.461666 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:48.461682 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.464289 (+ 2607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.464297 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:48.468351 (+ 4054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:48.468363 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.470130 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.470137 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:48.470305 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:48.470337 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 19:00:48.470444 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 19:00:48.470562 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1442,"mutex_wait_us":783,"thread_start_us":1314,"threads_started":1}
I20260924 19:00:48.470985 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:48.458675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52416 (local address 127.0.97.3:35721)
0924 19:00:48.459545 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:48.459553 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:48.460038 (+ 485us) server_negotiation.cc:407] Connection header received
0924 19:00:48.460128 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:48.460135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:48.460430 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:48.460634 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:48.461927 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.464079 (+ 2152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.468520 (+ 4441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.469937 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.470064 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:48.470292 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:48.470489 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 19:00:48.470642 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":450,"threads_started":1}
I20260924 19:00:48.472683 8119 tablet_service.cc:1467] Processing CreateTablet for tablet 18559084fb9e4e29a9706dad8d4ae029 (DEFAULT_TABLE table=test-table [id=3dafe0c7da4b40cd851761ce3ece0f21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:48.474143 8119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18559084fb9e4e29a9706dad8d4ae029. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:48.474726 8397 tablet_service.cc:1467] Processing CreateTablet for tablet 18559084fb9e4e29a9706dad8d4ae029 (DEFAULT_TABLE table=test-table [id=3dafe0c7da4b40cd851761ce3ece0f21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:48.475351 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:48.460091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55670 (local address 127.0.97.2:46793)
0924 19:00:48.468611 (+ 8520us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:48.468619 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:48.468645 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:00:48.468738 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:48.468747 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:48.469005 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:48.469226 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:48.469795 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.471188 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.473443 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.474384 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.474532 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:48.474727 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:48.474945 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 19:00:48.475094 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8357,"thread_start_us":8240,"threads_started":1}
I20260924 19:00:48.475562 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:48.460715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46793 (local address 127.0.0.1:55670)
0924 19:00:48.467595 (+ 6880us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:48.467618 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:48.467744 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:48.469218 (+ 1474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:48.469223 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:48.469247 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:48.469614 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:48.469626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.471340 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.471345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:48.473234 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:48.473245 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.474578 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.474584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:48.474737 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:48.474773 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 19:00:48.474890 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 19:00:48.475024 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6718,"thread_start_us":7377,"threads_started":1}
I20260924 19:00:48.476032 8397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18559084fb9e4e29a9706dad8d4ae029. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:48.479929 8258 tablet_service.cc:1467] Processing CreateTablet for tablet 18559084fb9e4e29a9706dad8d4ae029 (DEFAULT_TABLE table=test-table [id=3dafe0c7da4b40cd851761ce3ece0f21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:48.480970 8258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18559084fb9e4e29a9706dad8d4ae029. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:48.488431 8491 tablet_bootstrap.cc:492] T 18559084fb9e4e29a9706dad8d4ae029 P 934856ed75d54e2cbc38989df8f509d1: Bootstrap starting.
I20260924 19:00:48.490449 8492 tablet_bootstrap.cc:492] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b: Bootstrap starting.
I20260924 19:00:48.492064 8491 tablet_bootstrap.cc:654] T 18559084fb9e4e29a9706dad8d4ae029 P 934856ed75d54e2cbc38989df8f509d1: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:48.492147 8493 tablet_bootstrap.cc:492] T 18559084fb9e4e29a9706dad8d4ae029 P 69ea0ce743ff4c8ca80afa6c7da26941: Bootstrap starting.
I20260924 19:00:48.493726 8492 tablet_bootstrap.cc:654] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:48.494446 8491 log.cc:824] T 18559084fb9e4e29a9706dad8d4ae029 P 934856ed75d54e2cbc38989df8f509d1: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:48.494881 8492 log.cc:824] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:48.495253 8493 tablet_bootstrap.cc:654] T 18559084fb9e4e29a9706dad8d4ae029 P 69ea0ce743ff4c8ca80afa6c7da26941: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:48.496477 8493 log.cc:824] T 18559084fb9e4e29a9706dad8d4ae029 P 69ea0ce743ff4c8ca80afa6c7da26941: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:48.497656 8491 tablet_bootstrap.cc:492] T 18559084fb9e4e29a9706dad8d4ae029 P 934856ed75d54e2cbc38989df8f509d1: No bootstrap required, opened a new log
I20260924 19:00:48.497661 8492 tablet_bootstrap.cc:492] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b: No bootstrap required, opened a new log
I20260924 19:00:48.497916 8491 ts_tablet_manager.cc:1397] T 18559084fb9e4e29a9706dad8d4ae029 P 934856ed75d54e2cbc38989df8f509d1: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260924 19:00:48.497929 8492 ts_tablet_manager.cc:1397] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260924 19:00:48.499655 8493 tablet_bootstrap.cc:492] T 18559084fb9e4e29a9706dad8d4ae029 P 69ea0ce743ff4c8ca80afa6c7da26941: No bootstrap required, opened a new log
I20260924 19:00:48.499883 8493 ts_tablet_manager.cc:1397] T 18559084fb9e4e29a9706dad8d4ae029 P 69ea0ce743ff4c8ca80afa6c7da26941: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 19:00:48.505123 8492 raft_consensus.cc:348] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } }
I20260924 19:00:48.505482 8492 raft_consensus.cc:374] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:48.505646 8492 raft_consensus.cc:729] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e31a53118f4d45ad946d20cf4952a46b, State: Initialized, Role: FOLLOWER
I20260924 19:00:48.506240 8492 consensus_queue.cc:260] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } }
I20260924 19:00:48.507504 8467 heartbeater.cc:503] Master 127.0.97.62:38989 was elected leader, sending a full tablet report...
I20260924 19:00:48.508082 8492 ts_tablet_manager.cc:1428] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b: Time spent starting tablet: real 0.010s user 0.007s sys 0.003s
I20260924 19:00:48.508258 8491 raft_consensus.cc:348] T 18559084fb9e4e29a9706dad8d4ae029 P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } }
I20260924 19:00:48.508626 8491 raft_consensus.cc:374] T 18559084fb9e4e29a9706dad8d4ae029 P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:48.508734 8491 raft_consensus.cc:729] T 18559084fb9e4e29a9706dad8d4ae029 P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 934856ed75d54e2cbc38989df8f509d1, State: Initialized, Role: FOLLOWER
I20260924 19:00:48.509286 8491 consensus_queue.cc:260] T 18559084fb9e4e29a9706dad8d4ae029 P 934856ed75d54e2cbc38989df8f509d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } }
I20260924 19:00:48.512199 8493 raft_consensus.cc:348] T 18559084fb9e4e29a9706dad8d4ae029 P 69ea0ce743ff4c8ca80afa6c7da26941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } }
I20260924 19:00:48.512723 8493 raft_consensus.cc:374] T 18559084fb9e4e29a9706dad8d4ae029 P 69ea0ce743ff4c8ca80afa6c7da26941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:48.512835 8493 raft_consensus.cc:729] T 18559084fb9e4e29a9706dad8d4ae029 P 69ea0ce743ff4c8ca80afa6c7da26941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ea0ce743ff4c8ca80afa6c7da26941, State: Initialized, Role: FOLLOWER
I20260924 19:00:48.513671 8493 consensus_queue.cc:260] T 18559084fb9e4e29a9706dad8d4ae029 P 69ea0ce743ff4c8ca80afa6c7da26941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } }
I20260924 19:00:48.514799 8328 heartbeater.cc:503] Master 127.0.97.62:38989 was elected leader, sending a full tablet report...
I20260924 19:00:48.515722 8493 ts_tablet_manager.cc:1428] T 18559084fb9e4e29a9706dad8d4ae029 P 69ea0ce743ff4c8ca80afa6c7da26941: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s
I20260924 19:00:48.517174 8491 ts_tablet_manager.cc:1428] T 18559084fb9e4e29a9706dad8d4ae029 P 934856ed75d54e2cbc38989df8f509d1: Time spent starting tablet: real 0.019s user 0.016s sys 0.003s
W20260924 19:00:48.541062 8190 tablet.cc:2369] T 18559084fb9e4e29a9706dad8d4ae029 P 934856ed75d54e2cbc38989df8f509d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:00:48.585325 8329 tablet.cc:2369] T 18559084fb9e4e29a9706dad8d4ae029 P 69ea0ce743ff4c8ca80afa6c7da26941: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:00:48.617462 8468 tablet.cc:2369] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:00:48.655742 8497 raft_consensus.cc:482] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:00:48.656083 8497 raft_consensus.cc:504] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } }
I20260924 19:00:48.657878 8497 leader_election.cc:288] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 934856ed75d54e2cbc38989df8f509d1 (127.0.97.1:40683), 69ea0ce743ff4c8ca80afa6c7da26941 (127.0.97.2:46793)
I20260924 19:00:48.663372 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:48.658169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56123 (local address 127.0.97.1:40683)
0924 19:00:48.658360 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:48.658367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:48.658427 (+ 60us) server_negotiation.cc:407] Connection header received
0924 19:00:48.658722 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:48.658731 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:48.658840 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:48.658942 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:48.659440 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.660142 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.661775 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.662519 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.662660 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:48.662768 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:48.663026 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 19:00:48.663140 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 19:00:48.663406 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:48.658103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40683 (local address 127.0.97.3:56123)
0924 19:00:48.658347 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:48.658367 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:48.658540 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:48.659042 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:48.659047 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:48.659075 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:48.659254 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:48.659262 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.660288 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.660295 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:48.661627 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:48.661638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.662679 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.662685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:48.662877 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:48.662899 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 19:00:48.663002 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 19:00:48.663125 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 19:00:48.664191 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:48.658807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42919 (local address 127.0.97.2:46793)
0924 19:00:48.659008 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:48.659015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:48.659214 (+ 199us) server_negotiation.cc:407] Connection header received
0924 19:00:48.659426 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:48.659431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:48.659486 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:48.659580 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:48.660145 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.660827 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.662507 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.663235 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.663386 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:48.663499 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:48.663825 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 19:00:48.663942 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:00:48.664227 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:48.658533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46793 (local address 127.0.97.3:42919)
0924 19:00:48.659118 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:48.659134 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:48.659271 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:48.659594 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:48.659598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:48.659620 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:48.659983 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:48.659991 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.660970 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.660977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:48.662363 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:48.662373 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.663451 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.663457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:48.663658 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:48.663695 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 19:00:48.663792 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 19:00:48.663920 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":297,"threads_started":1}
I20260924 19:00:48.664336 8141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18559084fb9e4e29a9706dad8d4ae029" candidate_uuid: "e31a53118f4d45ad946d20cf4952a46b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "934856ed75d54e2cbc38989df8f509d1" is_pre_election: true
I20260924 19:00:48.664885 8141 raft_consensus.cc:2454] T 18559084fb9e4e29a9706dad8d4ae029 P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e31a53118f4d45ad946d20cf4952a46b in term 0.
I20260924 19:00:48.665406 8278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18559084fb9e4e29a9706dad8d4ae029" candidate_uuid: "e31a53118f4d45ad946d20cf4952a46b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" is_pre_election: true
I20260924 19:00:48.665566 8348 leader_election.cc:302] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 934856ed75d54e2cbc38989df8f509d1, e31a53118f4d45ad946d20cf4952a46b; no voters:
I20260924 19:00:48.666011 8497 raft_consensus.cc:2781] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:00:48.665988 8278 raft_consensus.cc:2454] T 18559084fb9e4e29a9706dad8d4ae029 P 69ea0ce743ff4c8ca80afa6c7da26941 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e31a53118f4d45ad946d20cf4952a46b in term 0.
I20260924 19:00:48.666152 8497 raft_consensus.cc:482] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:00:48.666256 8497 raft_consensus.cc:3037] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:48.668490 8497 raft_consensus.cc:504] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } }
I20260924 19:00:48.669173 8497 leader_election.cc:288] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b [CANDIDATE]: Term 1 election: Requested vote from peers 934856ed75d54e2cbc38989df8f509d1 (127.0.97.1:40683), 69ea0ce743ff4c8ca80afa6c7da26941 (127.0.97.2:46793)
I20260924 19:00:48.669631 8278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18559084fb9e4e29a9706dad8d4ae029" candidate_uuid: "e31a53118f4d45ad946d20cf4952a46b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ea0ce743ff4c8ca80afa6c7da26941"
I20260924 19:00:48.669660 8141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18559084fb9e4e29a9706dad8d4ae029" candidate_uuid: "e31a53118f4d45ad946d20cf4952a46b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "934856ed75d54e2cbc38989df8f509d1"
I20260924 19:00:48.669817 8278 raft_consensus.cc:3037] T 18559084fb9e4e29a9706dad8d4ae029 P 69ea0ce743ff4c8ca80afa6c7da26941 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:48.669834 8141 raft_consensus.cc:3037] T 18559084fb9e4e29a9706dad8d4ae029 P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:48.671872 8278 raft_consensus.cc:2454] T 18559084fb9e4e29a9706dad8d4ae029 P 69ea0ce743ff4c8ca80afa6c7da26941 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e31a53118f4d45ad946d20cf4952a46b in term 1.
I20260924 19:00:48.671883 8141 raft_consensus.cc:2454] T 18559084fb9e4e29a9706dad8d4ae029 P 934856ed75d54e2cbc38989df8f509d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e31a53118f4d45ad946d20cf4952a46b in term 1.
I20260924 19:00:48.672228 8348 leader_election.cc:302] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ea0ce743ff4c8ca80afa6c7da26941, e31a53118f4d45ad946d20cf4952a46b; no voters:
I20260924 19:00:48.672536 8497 raft_consensus.cc:2781] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:00:48.672976 8497 raft_consensus.cc:686] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b [term 1 LEADER]: Becoming Leader. State: Replica: e31a53118f4d45ad946d20cf4952a46b, State: Running, Role: LEADER
I20260924 19:00:48.673440 8497 consensus_queue.cc:237] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } }
I20260924 19:00:48.677769 7982 catalog_manager.cc:5495] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b reported cstate change: term changed from 0 to 1, leader changed from <none> to e31a53118f4d45ad946d20cf4952a46b (127.0.97.3). New cstate: current_term: 1 leader_uuid: "e31a53118f4d45ad946d20cf4952a46b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } health_report { overall_health: UNKNOWN } } }
I20260924 19:00:48.711378 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:48.705276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52420 (local address 127.0.97.3:35721)
0924 19:00:48.705473 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:48.705480 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:48.705626 (+ 146us) server_negotiation.cc:407] Connection header received
0924 19:00:48.705880 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:48.705884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:48.705953 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:48.706094 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:00:48.706455 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.707527 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.708997 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:48.709668 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:48.709807 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:48.710167 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:00:48.710841 (+ 674us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:00:48.710965 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:48.711028 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 19:00:48.711120 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 19:00:48.720221 8278 raft_consensus.cc:1261] T 18559084fb9e4e29a9706dad8d4ae029 P 69ea0ce743ff4c8ca80afa6c7da26941 [term 1 FOLLOWER]: Refusing update from remote peer e31a53118f4d45ad946d20cf4952a46b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:00:48.720219 8141 raft_consensus.cc:1261] T 18559084fb9e4e29a9706dad8d4ae029 P 934856ed75d54e2cbc38989df8f509d1 [term 1 FOLLOWER]: Refusing update from remote peer e31a53118f4d45ad946d20cf4952a46b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:00:48.721200 8497 consensus_queue.cc:1035] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b [LEADER]: Connected to new peer: Peer: permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:00:48.721565 8501 consensus_queue.cc:1035] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:00:48.733771 8504 mvcc.cc:205] Tried to move back new op lower bound from 7332972333947944960 to 7332972333770141696. Current Snapshot: MvccSnapshot[applied={T|T < 7332972333947944960}]
I20260924 19:00:48.746675 7981 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:00:48.746950 7981 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:00:48.750249 7981 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a099787801042bcb43901bb50891277: Sending DeleteTablet for 3 replicas of tablet 18559084fb9e4e29a9706dad8d4ae029
I20260924 19:00:48.751336 8119 tablet_service.cc:1514] Processing DeleteTablet for tablet 18559084fb9e4e29a9706dad8d4ae029 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:48 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48440
I20260924 19:00:48.751487 8397 tablet_service.cc:1514] Processing DeleteTablet for tablet 18559084fb9e4e29a9706dad8d4ae029 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:48 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52416
I20260924 19:00:48.751623 8258 tablet_service.cc:1514] Processing DeleteTablet for tablet 18559084fb9e4e29a9706dad8d4ae029 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:48 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55670
I20260924 19:00:48.752638 7981 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:00:48.753451 7981 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:00:48.763620 8118 tablet_service.cc:1467] Processing CreateTablet for tablet a700959c31824364a40153e541cf2924 (DEFAULT_TABLE table=test-table [id=913a43dd3aae4452b123fea2b7acc27a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:48.763841 8257 tablet_service.cc:1467] Processing CreateTablet for tablet a700959c31824364a40153e541cf2924 (DEFAULT_TABLE table=test-table [id=913a43dd3aae4452b123fea2b7acc27a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:48.764227 8118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a700959c31824364a40153e541cf2924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:48.764392 8257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a700959c31824364a40153e541cf2924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:48.770221 8396 tablet_service.cc:1467] Processing CreateTablet for tablet a700959c31824364a40153e541cf2924 (DEFAULT_TABLE table=test-table [id=913a43dd3aae4452b123fea2b7acc27a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:48.770855 8396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a700959c31824364a40153e541cf2924. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:48.771027 8493 tablet_bootstrap.cc:492] T a700959c31824364a40153e541cf2924 P 69ea0ce743ff4c8ca80afa6c7da26941: Bootstrap starting.
I20260924 19:00:48.772675 8493 tablet_bootstrap.cc:654] T a700959c31824364a40153e541cf2924 P 69ea0ce743ff4c8ca80afa6c7da26941: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:48.773986 8515 tablet_replica.cc:330] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b: stopping tablet replica
I20260924 19:00:48.774467 8515 raft_consensus.cc:2229] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:00:48.775132 8515 raft_consensus.cc:2258] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:48.775925 8514 tablet_replica.cc:330] T 18559084fb9e4e29a9706dad8d4ae029 P 934856ed75d54e2cbc38989df8f509d1: stopping tablet replica
I20260924 19:00:48.776350 8514 raft_consensus.cc:2229] T 18559084fb9e4e29a9706dad8d4ae029 P 934856ed75d54e2cbc38989df8f509d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:00:48.776851 8514 raft_consensus.cc:2258] T 18559084fb9e4e29a9706dad8d4ae029 P 934856ed75d54e2cbc38989df8f509d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:48.777626 8516 tablet_replica.cc:330] T 18559084fb9e4e29a9706dad8d4ae029 P 69ea0ce743ff4c8ca80afa6c7da26941: stopping tablet replica
I20260924 19:00:48.777987 8516 raft_consensus.cc:2229] T 18559084fb9e4e29a9706dad8d4ae029 P 69ea0ce743ff4c8ca80afa6c7da26941 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:00:48.778441 8516 raft_consensus.cc:2258] T 18559084fb9e4e29a9706dad8d4ae029 P 69ea0ce743ff4c8ca80afa6c7da26941 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:48.781697 8516 ts_tablet_manager.cc:1905] T 18559084fb9e4e29a9706dad8d4ae029 P 69ea0ce743ff4c8ca80afa6c7da26941: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:48.783119 8493 tablet_bootstrap.cc:492] T a700959c31824364a40153e541cf2924 P 69ea0ce743ff4c8ca80afa6c7da26941: No bootstrap required, opened a new log
I20260924 19:00:48.783268 8493 ts_tablet_manager.cc:1397] T a700959c31824364a40153e541cf2924 P 69ea0ce743ff4c8ca80afa6c7da26941: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.011s
I20260924 19:00:48.784153 8493 raft_consensus.cc:348] T a700959c31824364a40153e541cf2924 P 69ea0ce743ff4c8ca80afa6c7da26941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:48.784387 8493 raft_consensus.cc:374] T a700959c31824364a40153e541cf2924 P 69ea0ce743ff4c8ca80afa6c7da26941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:48.784493 8493 raft_consensus.cc:729] T a700959c31824364a40153e541cf2924 P 69ea0ce743ff4c8ca80afa6c7da26941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ea0ce743ff4c8ca80afa6c7da26941, State: Initialized, Role: FOLLOWER
I20260924 19:00:48.784811 8493 consensus_queue.cc:260] T a700959c31824364a40153e541cf2924 P 69ea0ce743ff4c8ca80afa6c7da26941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:48.785388 8493 ts_tablet_manager.cc:1428] T a700959c31824364a40153e541cf2924 P 69ea0ce743ff4c8ca80afa6c7da26941: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 19:00:48.788691 8516 ts_tablet_manager.cc:1918] T 18559084fb9e4e29a9706dad8d4ae029 P 69ea0ce743ff4c8ca80afa6c7da26941: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:00:48.788867 8516 log.cc:1196] T 18559084fb9e4e29a9706dad8d4ae029 P 69ea0ce743ff4c8ca80afa6c7da26941: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-1/wal/wals/18559084fb9e4e29a9706dad8d4ae029
I20260924 19:00:48.789412 8492 tablet_bootstrap.cc:492] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b: Bootstrap starting.
I20260924 19:00:48.790128 8491 tablet_bootstrap.cc:492] T a700959c31824364a40153e541cf2924 P 934856ed75d54e2cbc38989df8f509d1: Bootstrap starting.
I20260924 19:00:48.791149 8492 tablet_bootstrap.cc:654] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:48.791674 8491 tablet_bootstrap.cc:654] T a700959c31824364a40153e541cf2924 P 934856ed75d54e2cbc38989df8f509d1: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:48.798048 8516 ts_tablet_manager.cc:1939] T 18559084fb9e4e29a9706dad8d4ae029 P 69ea0ce743ff4c8ca80afa6c7da26941: Deleting consensus metadata
I20260924 19:00:48.799048 8492 tablet_bootstrap.cc:492] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b: No bootstrap required, opened a new log
I20260924 19:00:48.799207 8492 ts_tablet_manager.cc:1397] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20260924 19:00:48.800092 8492 raft_consensus.cc:348] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:48.800330 8492 raft_consensus.cc:374] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:48.800431 8492 raft_consensus.cc:729] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e31a53118f4d45ad946d20cf4952a46b, State: Initialized, Role: FOLLOWER
I20260924 19:00:48.800709 8492 consensus_queue.cc:260] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:48.801779 8492 ts_tablet_manager.cc:1428] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 19:00:48.803587 7966 catalog_manager.cc:4841] TS 69ea0ce743ff4c8ca80afa6c7da26941 (127.0.97.2:46793): tablet 18559084fb9e4e29a9706dad8d4ae029 (table test-table [id=3dafe0c7da4b40cd851761ce3ece0f21]) successfully deleted
I20260924 19:00:48.805238 8491 tablet_bootstrap.cc:492] T a700959c31824364a40153e541cf2924 P 934856ed75d54e2cbc38989df8f509d1: No bootstrap required, opened a new log
I20260924 19:00:48.805384 8491 ts_tablet_manager.cc:1397] T a700959c31824364a40153e541cf2924 P 934856ed75d54e2cbc38989df8f509d1: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.013s
I20260924 19:00:48.806174 8491 raft_consensus.cc:348] T a700959c31824364a40153e541cf2924 P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:48.806399 8491 raft_consensus.cc:374] T a700959c31824364a40153e541cf2924 P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:48.806491 8491 raft_consensus.cc:729] T a700959c31824364a40153e541cf2924 P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 934856ed75d54e2cbc38989df8f509d1, State: Initialized, Role: FOLLOWER
I20260924 19:00:48.806741 8491 consensus_queue.cc:260] T a700959c31824364a40153e541cf2924 P 934856ed75d54e2cbc38989df8f509d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:48.807296 8491 ts_tablet_manager.cc:1428] T a700959c31824364a40153e541cf2924 P 934856ed75d54e2cbc38989df8f509d1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 19:00:48.807852 8514 ts_tablet_manager.cc:1905] T 18559084fb9e4e29a9706dad8d4ae029 P 934856ed75d54e2cbc38989df8f509d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:48.809093 8515 ts_tablet_manager.cc:1905] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:48.814819 8514 ts_tablet_manager.cc:1918] T 18559084fb9e4e29a9706dad8d4ae029 P 934856ed75d54e2cbc38989df8f509d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:00:48.814976 8514 log.cc:1196] T 18559084fb9e4e29a9706dad8d4ae029 P 934856ed75d54e2cbc38989df8f509d1: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-0/wal/wals/18559084fb9e4e29a9706dad8d4ae029
I20260924 19:00:48.815384 8515 ts_tablet_manager.cc:1918] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:00:48.815493 8514 ts_tablet_manager.cc:1939] T 18559084fb9e4e29a9706dad8d4ae029 P 934856ed75d54e2cbc38989df8f509d1: Deleting consensus metadata
I20260924 19:00:48.815589 8515 log.cc:1196] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-2/wal/wals/18559084fb9e4e29a9706dad8d4ae029
I20260924 19:00:48.816067 8515 ts_tablet_manager.cc:1939] T 18559084fb9e4e29a9706dad8d4ae029 P e31a53118f4d45ad946d20cf4952a46b: Deleting consensus metadata
I20260924 19:00:48.817222 7966 catalog_manager.cc:4841] TS 934856ed75d54e2cbc38989df8f509d1 (127.0.97.1:40683): tablet 18559084fb9e4e29a9706dad8d4ae029 (table test-table [id=3dafe0c7da4b40cd851761ce3ece0f21]) successfully deleted
I20260924 19:00:48.817487 7966 catalog_manager.cc:4841] TS e31a53118f4d45ad946d20cf4952a46b (127.0.97.3:35721): tablet 18559084fb9e4e29a9706dad8d4ae029 (table test-table [id=3dafe0c7da4b40cd851761ce3ece0f21]) successfully deleted
I20260924 19:00:49.034447 8509 raft_consensus.cc:482] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:00:49.034654 8509 raft_consensus.cc:504] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:49.035403 8509 leader_election.cc:288] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 934856ed75d54e2cbc38989df8f509d1 (127.0.97.1:40683), 69ea0ce743ff4c8ca80afa6c7da26941 (127.0.97.2:46793)
I20260924 19:00:49.035912 8141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a700959c31824364a40153e541cf2924" candidate_uuid: "e31a53118f4d45ad946d20cf4952a46b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "934856ed75d54e2cbc38989df8f509d1" is_pre_election: true
I20260924 19:00:49.036028 8278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a700959c31824364a40153e541cf2924" candidate_uuid: "e31a53118f4d45ad946d20cf4952a46b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" is_pre_election: true
I20260924 19:00:49.036168 8141 raft_consensus.cc:2454] T a700959c31824364a40153e541cf2924 P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e31a53118f4d45ad946d20cf4952a46b in term 0.
I20260924 19:00:49.036329 8278 raft_consensus.cc:2454] T a700959c31824364a40153e541cf2924 P 69ea0ce743ff4c8ca80afa6c7da26941 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e31a53118f4d45ad946d20cf4952a46b in term 0.
I20260924 19:00:49.036733 8348 leader_election.cc:302] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 934856ed75d54e2cbc38989df8f509d1, e31a53118f4d45ad946d20cf4952a46b; no voters:
I20260924 19:00:49.037036 8509 raft_consensus.cc:2781] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:00:49.037164 8509 raft_consensus.cc:482] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:00:49.037273 8509 raft_consensus.cc:3037] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:49.040460 8509 raft_consensus.cc:504] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:49.041138 8509 leader_election.cc:288] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b [CANDIDATE]: Term 1 election: Requested vote from peers 934856ed75d54e2cbc38989df8f509d1 (127.0.97.1:40683), 69ea0ce743ff4c8ca80afa6c7da26941 (127.0.97.2:46793)
I20260924 19:00:49.041600 8141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a700959c31824364a40153e541cf2924" candidate_uuid: "e31a53118f4d45ad946d20cf4952a46b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "934856ed75d54e2cbc38989df8f509d1"
I20260924 19:00:49.041725 8278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a700959c31824364a40153e541cf2924" candidate_uuid: "e31a53118f4d45ad946d20cf4952a46b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ea0ce743ff4c8ca80afa6c7da26941"
I20260924 19:00:49.041822 8141 raft_consensus.cc:3037] T a700959c31824364a40153e541cf2924 P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:49.041924 8278 raft_consensus.cc:3037] T a700959c31824364a40153e541cf2924 P 69ea0ce743ff4c8ca80afa6c7da26941 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:49.044739 8141 raft_consensus.cc:2454] T a700959c31824364a40153e541cf2924 P 934856ed75d54e2cbc38989df8f509d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e31a53118f4d45ad946d20cf4952a46b in term 1.
I20260924 19:00:49.044739 8278 raft_consensus.cc:2454] T a700959c31824364a40153e541cf2924 P 69ea0ce743ff4c8ca80afa6c7da26941 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e31a53118f4d45ad946d20cf4952a46b in term 1.
I20260924 19:00:49.045183 8348 leader_election.cc:302] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ea0ce743ff4c8ca80afa6c7da26941, e31a53118f4d45ad946d20cf4952a46b; no voters:
I20260924 19:00:49.045796 8509 raft_consensus.cc:2781] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:00:49.045996 8509 raft_consensus.cc:686] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b [term 1 LEADER]: Becoming Leader. State: Replica: e31a53118f4d45ad946d20cf4952a46b, State: Running, Role: LEADER
I20260924 19:00:49.046277 8509 consensus_queue.cc:237] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:49.049489 7982 catalog_manager.cc:5495] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b reported cstate change: term changed from 0 to 1, leader changed from <none> to e31a53118f4d45ad946d20cf4952a46b (127.0.97.3). New cstate: current_term: 1 leader_uuid: "e31a53118f4d45ad946d20cf4952a46b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } health_report { overall_health: HEALTHY } } }
Sep 24 19:00:49 dist-test-slave-9fn1 krb5kdc[7937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276446, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 19:00:49.117588 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:49.103887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53332 (local address 127.0.97.62:38989)
0924 19:00:49.104540 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:49.104547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:49.104773 (+ 226us) server_negotiation.cc:407] Connection header received
0924 19:00:49.105020 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:49.105023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:49.105083 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:49.105189 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:49.106295 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:49.107052 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:49.108337 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:49.108774 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:49.108927 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:49.112416 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:49.112422 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:49.112432 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:49.112458 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:49.114990 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:49.115627 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:49.115632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:49.115635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:49.115744 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:49.116119 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:49.116127 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:49.116130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:49.116372 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:49.116532 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:49.116920 (+ 388us) server_negotiation.cc:299] Negotiation successful
0924 19:00:49.117204 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":462,"thread_start_us":305,"threads_started":1}
I20260924 19:00:49.117561 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:49.103886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38989 (local address 127.0.0.1:53332)
0924 19:00:49.104671 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:49.104690 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:49.104878 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:49.105211 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:49.105215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:49.105971 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:49.106161 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:49.106169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:49.107189 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:49.107196 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:49.108188 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:49.108199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:49.108939 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:49.108944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:49.109192 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:49.109905 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:49.109924 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:49.112223 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:49.115117 (+ 2894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:49.115128 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:49.115135 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:49.115505 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:49.115852 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:49.115856 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:49.115860 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:49.116005 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:49.116500 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:49.116510 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:49.116662 (+ 152us) client_negotiation.cc:769] Sending connection context
0924 19:00:49.116825 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 19:00:49.117100 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":40,"thread_start_us":526,"threads_started":1}
I20260924 19:00:49.123126 7982 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53332:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 19:00:49.124313 7982 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:00:49.131183 8118 tablet_service.cc:1467] Processing CreateTablet for tablet 59c099de6c58413a988d6e98cda8f71c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=80224459a5984a0ba9e322327802b935]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:00:49.131181 8258 tablet_service.cc:1467] Processing CreateTablet for tablet 59c099de6c58413a988d6e98cda8f71c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=80224459a5984a0ba9e322327802b935]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:00:49.131438 8396 tablet_service.cc:1467] Processing CreateTablet for tablet 59c099de6c58413a988d6e98cda8f71c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=80224459a5984a0ba9e322327802b935]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:00:49.131690 8118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59c099de6c58413a988d6e98cda8f71c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:49.131754 8258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59c099de6c58413a988d6e98cda8f71c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:49.131893 8396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59c099de6c58413a988d6e98cda8f71c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:49.138906 8491 tablet_bootstrap.cc:492] T 59c099de6c58413a988d6e98cda8f71c P 934856ed75d54e2cbc38989df8f509d1: Bootstrap starting.
I20260924 19:00:49.139171 8492 tablet_bootstrap.cc:492] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b: Bootstrap starting.
I20260924 19:00:49.140581 8491 tablet_bootstrap.cc:654] T 59c099de6c58413a988d6e98cda8f71c P 934856ed75d54e2cbc38989df8f509d1: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:49.140591 8492 tablet_bootstrap.cc:654] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:49.141554 8493 tablet_bootstrap.cc:492] T 59c099de6c58413a988d6e98cda8f71c P 69ea0ce743ff4c8ca80afa6c7da26941: Bootstrap starting.
I20260924 19:00:49.142478 8492 tablet_bootstrap.cc:492] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b: No bootstrap required, opened a new log
I20260924 19:00:49.142663 8492 ts_tablet_manager.cc:1397] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260924 19:00:49.143074 8493 tablet_bootstrap.cc:654] T 59c099de6c58413a988d6e98cda8f71c P 69ea0ce743ff4c8ca80afa6c7da26941: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:49.143431 8492 raft_consensus.cc:348] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:49.143677 8492 raft_consensus.cc:374] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:49.143761 8492 raft_consensus.cc:729] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e31a53118f4d45ad946d20cf4952a46b, State: Initialized, Role: FOLLOWER
I20260924 19:00:49.144042 8492 consensus_queue.cc:260] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:49.144454 8509 tablet_replica.cc:439] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } } }
I20260924 19:00:49.144717 8509 tablet_replica.cc:442] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:00:49.145592 8492 ts_tablet_manager.cc:1428] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 19:00:49.147928 8491 tablet_bootstrap.cc:492] T 59c099de6c58413a988d6e98cda8f71c P 934856ed75d54e2cbc38989df8f509d1: No bootstrap required, opened a new log
I20260924 19:00:49.148092 8491 ts_tablet_manager.cc:1397] T 59c099de6c58413a988d6e98cda8f71c P 934856ed75d54e2cbc38989df8f509d1: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260924 19:00:49.149113 8491 raft_consensus.cc:348] T 59c099de6c58413a988d6e98cda8f71c P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:49.149361 8491 raft_consensus.cc:374] T 59c099de6c58413a988d6e98cda8f71c P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:49.149464 8491 raft_consensus.cc:729] T 59c099de6c58413a988d6e98cda8f71c P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 934856ed75d54e2cbc38989df8f509d1, State: Initialized, Role: FOLLOWER
I20260924 19:00:49.149722 8491 consensus_queue.cc:260] T 59c099de6c58413a988d6e98cda8f71c P 934856ed75d54e2cbc38989df8f509d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:49.150295 8491 ts_tablet_manager.cc:1428] T 59c099de6c58413a988d6e98cda8f71c P 934856ed75d54e2cbc38989df8f509d1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:00:49.150177 8498 tablet_replica.cc:439] T 59c099de6c58413a988d6e98cda8f71c P 934856ed75d54e2cbc38989df8f509d1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } } }
I20260924 19:00:49.150540 8498 tablet_replica.cc:442] T 59c099de6c58413a988d6e98cda8f71c P 934856ed75d54e2cbc38989df8f509d1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:00:49.154296 8493 tablet_bootstrap.cc:492] T 59c099de6c58413a988d6e98cda8f71c P 69ea0ce743ff4c8ca80afa6c7da26941: No bootstrap required, opened a new log
I20260924 19:00:49.154460 8493 ts_tablet_manager.cc:1397] T 59c099de6c58413a988d6e98cda8f71c P 69ea0ce743ff4c8ca80afa6c7da26941: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260924 19:00:49.155133 8493 raft_consensus.cc:348] T 59c099de6c58413a988d6e98cda8f71c P 69ea0ce743ff4c8ca80afa6c7da26941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:49.155362 8493 raft_consensus.cc:374] T 59c099de6c58413a988d6e98cda8f71c P 69ea0ce743ff4c8ca80afa6c7da26941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:49.155433 8493 raft_consensus.cc:729] T 59c099de6c58413a988d6e98cda8f71c P 69ea0ce743ff4c8ca80afa6c7da26941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ea0ce743ff4c8ca80afa6c7da26941, State: Initialized, Role: FOLLOWER
I20260924 19:00:49.155656 8493 consensus_queue.cc:260] T 59c099de6c58413a988d6e98cda8f71c P 69ea0ce743ff4c8ca80afa6c7da26941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:49.156158 8499 tablet_replica.cc:439] T 59c099de6c58413a988d6e98cda8f71c P 69ea0ce743ff4c8ca80afa6c7da26941: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } } }
I20260924 19:00:49.156509 8499 tablet_replica.cc:442] T 59c099de6c58413a988d6e98cda8f71c P 69ea0ce743ff4c8ca80afa6c7da26941: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:00:49.156965 8493 ts_tablet_manager.cc:1428] T 59c099de6c58413a988d6e98cda8f71c P 69ea0ce743ff4c8ca80afa6c7da26941: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 19:00:49.280833 8509 raft_consensus.cc:482] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:00:49.281085 8509 raft_consensus.cc:504] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:49.281720 8509 leader_election.cc:288] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 934856ed75d54e2cbc38989df8f509d1 (127.0.97.1:40683), 69ea0ce743ff4c8ca80afa6c7da26941 (127.0.97.2:46793)
I20260924 19:00:49.282383 8141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59c099de6c58413a988d6e98cda8f71c" candidate_uuid: "e31a53118f4d45ad946d20cf4952a46b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "934856ed75d54e2cbc38989df8f509d1" is_pre_election: true
I20260924 19:00:49.282429 8278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59c099de6c58413a988d6e98cda8f71c" candidate_uuid: "e31a53118f4d45ad946d20cf4952a46b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" is_pre_election: true
I20260924 19:00:49.282671 8141 raft_consensus.cc:2454] T 59c099de6c58413a988d6e98cda8f71c P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e31a53118f4d45ad946d20cf4952a46b in term 0.
I20260924 19:00:49.282671 8278 raft_consensus.cc:2454] T 59c099de6c58413a988d6e98cda8f71c P 69ea0ce743ff4c8ca80afa6c7da26941 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e31a53118f4d45ad946d20cf4952a46b in term 0.
I20260924 19:00:49.283092 8348 leader_election.cc:302] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 934856ed75d54e2cbc38989df8f509d1, e31a53118f4d45ad946d20cf4952a46b; no voters:
I20260924 19:00:49.283365 8509 raft_consensus.cc:2781] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:00:49.283463 8509 raft_consensus.cc:482] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:00:49.283569 8509 raft_consensus.cc:3037] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:49.285611 8509 raft_consensus.cc:504] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:49.286209 8509 leader_election.cc:288] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b [CANDIDATE]: Term 1 election: Requested vote from peers 934856ed75d54e2cbc38989df8f509d1 (127.0.97.1:40683), 69ea0ce743ff4c8ca80afa6c7da26941 (127.0.97.2:46793)
I20260924 19:00:49.286715 8141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59c099de6c58413a988d6e98cda8f71c" candidate_uuid: "e31a53118f4d45ad946d20cf4952a46b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "934856ed75d54e2cbc38989df8f509d1"
I20260924 19:00:49.286749 8278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59c099de6c58413a988d6e98cda8f71c" candidate_uuid: "e31a53118f4d45ad946d20cf4952a46b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ea0ce743ff4c8ca80afa6c7da26941"
I20260924 19:00:49.286960 8141 raft_consensus.cc:3037] T 59c099de6c58413a988d6e98cda8f71c P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:49.286955 8278 raft_consensus.cc:3037] T 59c099de6c58413a988d6e98cda8f71c P 69ea0ce743ff4c8ca80afa6c7da26941 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:49.289774 8141 raft_consensus.cc:2454] T 59c099de6c58413a988d6e98cda8f71c P 934856ed75d54e2cbc38989df8f509d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e31a53118f4d45ad946d20cf4952a46b in term 1.
I20260924 19:00:49.289772 8278 raft_consensus.cc:2454] T 59c099de6c58413a988d6e98cda8f71c P 69ea0ce743ff4c8ca80afa6c7da26941 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e31a53118f4d45ad946d20cf4952a46b in term 1.
I20260924 19:00:49.290131 8348 leader_election.cc:302] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 934856ed75d54e2cbc38989df8f509d1, e31a53118f4d45ad946d20cf4952a46b; no voters:
I20260924 19:00:49.290407 8509 raft_consensus.cc:2781] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:00:49.290581 8509 raft_consensus.cc:686] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b [term 1 LEADER]: Becoming Leader. State: Replica: e31a53118f4d45ad946d20cf4952a46b, State: Running, Role: LEADER
I20260924 19:00:49.291075 8509 consensus_queue.cc:237] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:49.292595 8501 tablet_replica.cc:439] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b: TxnStatusTablet state changed. Reason: New leader e31a53118f4d45ad946d20cf4952a46b. Latest consensus state: current_term: 1 leader_uuid: "e31a53118f4d45ad946d20cf4952a46b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } } }
I20260924 19:00:49.292829 8501 tablet_replica.cc:442] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:00:49.293347 8532 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 19:00:49.293572 8532 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 19:00:49.294034 7981 catalog_manager.cc:5495] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b reported cstate change: term changed from 0 to 1, leader changed from <none> to e31a53118f4d45ad946d20cf4952a46b (127.0.97.3). New cstate: current_term: 1 leader_uuid: "e31a53118f4d45ad946d20cf4952a46b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } health_report { overall_health: HEALTHY } } }
I20260924 19:00:49.343804 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:49.337916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35721 (local address 127.0.0.1:52428)
0924 19:00:49.338138 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:49.338163 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:49.338299 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:49.339193 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:49.339197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:49.339207 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:00:49.339355 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:49.339362 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:49.340815 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:49.340824 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:49.341859 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:49.341870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:49.342814 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:49.342824 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:49.342957 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:49.343006 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:00:49.343401 (+ 395us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:00:49.343414 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 19:00:49.343472 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 19:00:49.343576 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 19:00:49.343952 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:49.337918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52428 (local address 127.0.97.3:35721)
0924 19:00:49.338853 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:49.338859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:49.338876 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:00:49.338945 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:49.338955 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:49.339041 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:49.339138 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:00:49.339505 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:49.340692 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:49.342014 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:49.342681 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:49.342814 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:49.343179 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:00:49.343300 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:00:49.343394 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:49.343516 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 19:00:49.343589 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":633,"threads_started":1}
I20260924 19:00:49.347512 8278 raft_consensus.cc:1261] T 59c099de6c58413a988d6e98cda8f71c P 69ea0ce743ff4c8ca80afa6c7da26941 [term 1 FOLLOWER]: Refusing update from remote peer e31a53118f4d45ad946d20cf4952a46b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:00:49.347771 8141 raft_consensus.cc:1261] T 59c099de6c58413a988d6e98cda8f71c P 934856ed75d54e2cbc38989df8f509d1 [term 1 FOLLOWER]: Refusing update from remote peer e31a53118f4d45ad946d20cf4952a46b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:00:49.348171 8501 consensus_queue.cc:1035] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:00:49.348613 8509 consensus_queue.cc:1035] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b [LEADER]: Connected to new peer: Peer: permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:00:49.354420 8498 tablet_replica.cc:439] T 59c099de6c58413a988d6e98cda8f71c P 934856ed75d54e2cbc38989df8f509d1: TxnStatusTablet state changed. Reason: New leader e31a53118f4d45ad946d20cf4952a46b. Latest consensus state: current_term: 1 leader_uuid: "e31a53118f4d45ad946d20cf4952a46b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } } }
I20260924 19:00:49.354765 8498 tablet_replica.cc:442] T 59c099de6c58413a988d6e98cda8f71c P 934856ed75d54e2cbc38989df8f509d1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:00:49.358304 8509 tablet_replica.cc:439] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e31a53118f4d45ad946d20cf4952a46b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } } }
I20260924 19:00:49.358611 8509 tablet_replica.cc:442] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:00:49.378607 8499 tablet_replica.cc:439] T 59c099de6c58413a988d6e98cda8f71c P 69ea0ce743ff4c8ca80afa6c7da26941: TxnStatusTablet state changed. Reason: New leader e31a53118f4d45ad946d20cf4952a46b. Latest consensus state: current_term: 1 leader_uuid: "e31a53118f4d45ad946d20cf4952a46b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } } }
I20260924 19:00:49.378906 8499 tablet_replica.cc:442] T 59c099de6c58413a988d6e98cda8f71c P 69ea0ce743ff4c8ca80afa6c7da26941: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:00:49.382001 8501 tablet_replica.cc:439] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e31a53118f4d45ad946d20cf4952a46b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } } }
I20260924 19:00:49.382296 8501 tablet_replica.cc:442] T 59c099de6c58413a988d6e98cda8f71c P e31a53118f4d45ad946d20cf4952a46b: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:00:49.390709 8498 tablet_replica.cc:439] T 59c099de6c58413a988d6e98cda8f71c P 934856ed75d54e2cbc38989df8f509d1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e31a53118f4d45ad946d20cf4952a46b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } } }
I20260924 19:00:49.390956 8498 tablet_replica.cc:442] T 59c099de6c58413a988d6e98cda8f71c P 934856ed75d54e2cbc38989df8f509d1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:00:49.400115 8499 tablet_replica.cc:439] T 59c099de6c58413a988d6e98cda8f71c P 69ea0ce743ff4c8ca80afa6c7da26941: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e31a53118f4d45ad946d20cf4952a46b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } } }
I20260924 19:00:49.400368 8499 tablet_replica.cc:442] T 59c099de6c58413a988d6e98cda8f71c P 69ea0ce743ff4c8ca80afa6c7da26941: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:00:49.412637 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:49.399109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37615 (local address 127.0.97.3:35721)
0924 19:00:49.399270 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:49.399277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:49.406160 (+ 6883us) server_negotiation.cc:407] Connection header received
0924 19:00:49.406360 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:49.406365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:49.406439 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:49.406530 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:00:49.407151 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:49.408332 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:49.410606 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:49.411227 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:49.411349 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:49.411868 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:00:49.411986 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:00:49.412094 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:49.412281 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 19:00:49.412369 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 19:00:49.413014 8546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:49.398673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35721 (local address 127.0.97.3:37615)
0924 19:00:49.406040 (+ 7367us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:49.406062 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:49.406225 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:49.406576 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:49.406583 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:49.406595 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:00:49.406996 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:49.407008 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:49.408476 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:49.408482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:49.410456 (+ 1974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:49.410471 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:49.411404 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:49.411416 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:49.411613 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:49.411689 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:00:49.412132 (+ 443us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:00:49.412150 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 19:00:49.412239 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 19:00:49.412376 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7165,"thread_start_us":7015,"threads_started":1}
I20260924 19:00:49.423032 8277 raft_consensus.cc:1261] T a700959c31824364a40153e541cf2924 P 69ea0ce743ff4c8ca80afa6c7da26941 [term 1 FOLLOWER]: Refusing update from remote peer e31a53118f4d45ad946d20cf4952a46b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:00:49.423097 8141 raft_consensus.cc:1261] T a700959c31824364a40153e541cf2924 P 934856ed75d54e2cbc38989df8f509d1 [term 1 FOLLOWER]: Refusing update from remote peer e31a53118f4d45ad946d20cf4952a46b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:00:49.423550 8509 consensus_queue.cc:1035] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:00:49.423833 8501 consensus_queue.cc:1035] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b [LEADER]: Connected to new peer: Peer: permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:00:49.495317 7981 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:00:49.495530 7981 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:00:49.498586 7981 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a099787801042bcb43901bb50891277: Sending DeleteTablet for 3 replicas of tablet a700959c31824364a40153e541cf2924
I20260924 19:00:49.499655 8258 tablet_service.cc:1514] Processing DeleteTablet for tablet a700959c31824364a40153e541cf2924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:49 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55670
I20260924 19:00:49.499858 8118 tablet_service.cc:1514] Processing DeleteTablet for tablet a700959c31824364a40153e541cf2924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:49 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48440
I20260924 19:00:49.502960 8396 tablet_service.cc:1514] Processing DeleteTablet for tablet a700959c31824364a40153e541cf2924 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:49 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52416
I20260924 19:00:49.512344 8553 tablet_replica.cc:330] T a700959c31824364a40153e541cf2924 P 69ea0ce743ff4c8ca80afa6c7da26941: stopping tablet replica
I20260924 19:00:49.512547 8553 raft_consensus.cc:2229] T a700959c31824364a40153e541cf2924 P 69ea0ce743ff4c8ca80afa6c7da26941 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:00:49.512825 8553 raft_consensus.cc:2258] T a700959c31824364a40153e541cf2924 P 69ea0ce743ff4c8ca80afa6c7da26941 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:49.514736 8553 ts_tablet_manager.cc:1905] T a700959c31824364a40153e541cf2924 P 69ea0ce743ff4c8ca80afa6c7da26941: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:49.518006 8554 tablet_replica.cc:330] T a700959c31824364a40153e541cf2924 P 934856ed75d54e2cbc38989df8f509d1: stopping tablet replica
I20260924 19:00:49.518242 8554 raft_consensus.cc:2229] T a700959c31824364a40153e541cf2924 P 934856ed75d54e2cbc38989df8f509d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:00:49.518539 8554 raft_consensus.cc:2258] T a700959c31824364a40153e541cf2924 P 934856ed75d54e2cbc38989df8f509d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:49.518771 8555 tablet_replica.cc:330] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b: stopping tablet replica
I20260924 19:00:49.519034 8555 raft_consensus.cc:2229] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:00:49.519516 8555 raft_consensus.cc:2258] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:49.519665 8554 ts_tablet_manager.cc:1905] T a700959c31824364a40153e541cf2924 P 934856ed75d54e2cbc38989df8f509d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:49.520627 8555 ts_tablet_manager.cc:1905] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:49.521881 8553 ts_tablet_manager.cc:1918] T a700959c31824364a40153e541cf2924 P 69ea0ce743ff4c8ca80afa6c7da26941: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:00:49.522009 8553 log.cc:1196] T a700959c31824364a40153e541cf2924 P 69ea0ce743ff4c8ca80afa6c7da26941: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-1/wal/wals/a700959c31824364a40153e541cf2924
I20260924 19:00:49.522476 8553 ts_tablet_manager.cc:1939] T a700959c31824364a40153e541cf2924 P 69ea0ce743ff4c8ca80afa6c7da26941: Deleting consensus metadata
I20260924 19:00:49.523576 7966 catalog_manager.cc:4841] TS 69ea0ce743ff4c8ca80afa6c7da26941 (127.0.97.2:46793): tablet a700959c31824364a40153e541cf2924 (table test-table [id=913a43dd3aae4452b123fea2b7acc27a]) successfully deleted
I20260924 19:00:49.526423 8554 ts_tablet_manager.cc:1918] T a700959c31824364a40153e541cf2924 P 934856ed75d54e2cbc38989df8f509d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:00:49.526589 8554 log.cc:1196] T a700959c31824364a40153e541cf2924 P 934856ed75d54e2cbc38989df8f509d1: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-0/wal/wals/a700959c31824364a40153e541cf2924
I20260924 19:00:49.527042 8554 ts_tablet_manager.cc:1939] T a700959c31824364a40153e541cf2924 P 934856ed75d54e2cbc38989df8f509d1: Deleting consensus metadata
I20260924 19:00:49.528117 7966 catalog_manager.cc:4841] TS 934856ed75d54e2cbc38989df8f509d1 (127.0.97.1:40683): tablet a700959c31824364a40153e541cf2924 (table test-table [id=913a43dd3aae4452b123fea2b7acc27a]) successfully deleted
I20260924 19:00:49.528586 8555 ts_tablet_manager.cc:1918] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:00:49.528744 8555 log.cc:1196] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-2/wal/wals/a700959c31824364a40153e541cf2924
I20260924 19:00:49.529134 8555 ts_tablet_manager.cc:1939] T a700959c31824364a40153e541cf2924 P e31a53118f4d45ad946d20cf4952a46b: Deleting consensus metadata
I20260924 19:00:49.530126 7966 catalog_manager.cc:4841] TS e31a53118f4d45ad946d20cf4952a46b (127.0.97.3:35721): tablet a700959c31824364a40153e541cf2924 (table test-table [id=913a43dd3aae4452b123fea2b7acc27a]) successfully deleted
Sep 24 19:00:49 dist-test-slave-9fn1 krb5kdc[7937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 19:00:49 dist-test-slave-9fn1 krb5kdc[7937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:00:49 dist-test-slave-9fn1 krb5kdc[7937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 19:00:49.570089 8526 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:00:49.559842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53340 (local address 127.0.97.62:38989)
0924 19:00:49.560039 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:49.560048 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:49.560393 (+ 345us) server_negotiation.cc:407] Connection header received
0924 19:00:49.560630 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:49.560634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:49.560698 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:49.560805 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:49.562224 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:49.562924 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:49.564145 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:49.564571 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:49.564711 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:49.569826 (+ 5115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53340: BlockingRecv error: recv got EOF from 127.0.0.1:53340 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 19:00:49 dist-test-slave-9fn1 krb5kdc[7937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260924 19:00:49.587994 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:49.572913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53344 (local address 127.0.97.62:38989)
0924 19:00:49.573086 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:49.573093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:49.573971 (+ 878us) server_negotiation.cc:407] Connection header received
0924 19:00:49.574099 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:49.574109 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:49.574182 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:49.574287 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:49.575346 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:49.576419 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:49.577701 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:49.578327 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:49.578457 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:49.582008 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:49.582013 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:49.582018 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:49.582055 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:49.585259 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:49.585932 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:49.585938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:49.585942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:49.586092 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:49.586508 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:49.586514 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:49.586517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:49.586862 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:49.587108 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:49.587378 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 19:00:49.587616 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 19:00:49.591611 7981 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:00:49.592319 7981 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:00:49.599260 8118 tablet_service.cc:1467] Processing CreateTablet for tablet 019636196b2a4b518ce6b9383cba5312 (DEFAULT_TABLE table=test-table [id=6ab8bfa944af4d0f92b156a013951abc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:49.599737 8118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 019636196b2a4b518ce6b9383cba5312. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:49.601929 8258 tablet_service.cc:1467] Processing CreateTablet for tablet 019636196b2a4b518ce6b9383cba5312 (DEFAULT_TABLE table=test-table [id=6ab8bfa944af4d0f92b156a013951abc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:49.602355 8258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 019636196b2a4b518ce6b9383cba5312. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:49.602262 8396 tablet_service.cc:1467] Processing CreateTablet for tablet 019636196b2a4b518ce6b9383cba5312 (DEFAULT_TABLE table=test-table [id=6ab8bfa944af4d0f92b156a013951abc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:49.602690 8396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 019636196b2a4b518ce6b9383cba5312. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:49.608738 8493 tablet_bootstrap.cc:492] T 019636196b2a4b518ce6b9383cba5312 P 69ea0ce743ff4c8ca80afa6c7da26941: Bootstrap starting.
I20260924 19:00:49.608951 8492 tablet_bootstrap.cc:492] T 019636196b2a4b518ce6b9383cba5312 P e31a53118f4d45ad946d20cf4952a46b: Bootstrap starting.
I20260924 19:00:49.610348 8493 tablet_bootstrap.cc:654] T 019636196b2a4b518ce6b9383cba5312 P 69ea0ce743ff4c8ca80afa6c7da26941: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:49.610641 8492 tablet_bootstrap.cc:654] T 019636196b2a4b518ce6b9383cba5312 P e31a53118f4d45ad946d20cf4952a46b: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:49.611697 8491 tablet_bootstrap.cc:492] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1: Bootstrap starting.
I20260924 19:00:49.613359 8491 tablet_bootstrap.cc:654] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:49.615178 8492 tablet_bootstrap.cc:492] T 019636196b2a4b518ce6b9383cba5312 P e31a53118f4d45ad946d20cf4952a46b: No bootstrap required, opened a new log
I20260924 19:00:49.615435 8492 ts_tablet_manager.cc:1397] T 019636196b2a4b518ce6b9383cba5312 P e31a53118f4d45ad946d20cf4952a46b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 19:00:49.616513 8492 raft_consensus.cc:348] T 019636196b2a4b518ce6b9383cba5312 P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:49.616986 8492 raft_consensus.cc:374] T 019636196b2a4b518ce6b9383cba5312 P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:49.617161 8492 raft_consensus.cc:729] T 019636196b2a4b518ce6b9383cba5312 P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e31a53118f4d45ad946d20cf4952a46b, State: Initialized, Role: FOLLOWER
I20260924 19:00:49.617529 8492 consensus_queue.cc:260] T 019636196b2a4b518ce6b9383cba5312 P e31a53118f4d45ad946d20cf4952a46b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:49.620126 8492 ts_tablet_manager.cc:1428] T 019636196b2a4b518ce6b9383cba5312 P e31a53118f4d45ad946d20cf4952a46b: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260924 19:00:49.621666 8491 tablet_bootstrap.cc:492] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1: No bootstrap required, opened a new log
I20260924 19:00:49.621866 8491 ts_tablet_manager.cc:1397] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20260924 19:00:49.622673 8491 raft_consensus.cc:348] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:49.622982 8491 raft_consensus.cc:374] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:49.623128 8491 raft_consensus.cc:729] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 934856ed75d54e2cbc38989df8f509d1, State: Initialized, Role: FOLLOWER
I20260924 19:00:49.623221 8493 tablet_bootstrap.cc:492] T 019636196b2a4b518ce6b9383cba5312 P 69ea0ce743ff4c8ca80afa6c7da26941: No bootstrap required, opened a new log
I20260924 19:00:49.623375 8493 ts_tablet_manager.cc:1397] T 019636196b2a4b518ce6b9383cba5312 P 69ea0ce743ff4c8ca80afa6c7da26941: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.010s
I20260924 19:00:49.623507 8491 consensus_queue.cc:260] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:49.624264 8491 ts_tablet_manager.cc:1428] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:00:49.625849 8493 raft_consensus.cc:348] T 019636196b2a4b518ce6b9383cba5312 P 69ea0ce743ff4c8ca80afa6c7da26941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:49.626129 8493 raft_consensus.cc:374] T 019636196b2a4b518ce6b9383cba5312 P 69ea0ce743ff4c8ca80afa6c7da26941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:49.626231 8493 raft_consensus.cc:729] T 019636196b2a4b518ce6b9383cba5312 P 69ea0ce743ff4c8ca80afa6c7da26941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ea0ce743ff4c8ca80afa6c7da26941, State: Initialized, Role: FOLLOWER
I20260924 19:00:49.626487 8493 consensus_queue.cc:260] T 019636196b2a4b518ce6b9383cba5312 P 69ea0ce743ff4c8ca80afa6c7da26941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:49.627040 8493 ts_tablet_manager.cc:1428] T 019636196b2a4b518ce6b9383cba5312 P 69ea0ce743ff4c8ca80afa6c7da26941: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260924 19:00:49.798944 8498 raft_consensus.cc:482] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:00:49.799139 8498 raft_consensus.cc:504] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:49.800487 8498 leader_election.cc:288] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69ea0ce743ff4c8ca80afa6c7da26941 (127.0.97.2:46793), e31a53118f4d45ad946d20cf4952a46b (127.0.97.3:35721)
I20260924 19:00:49.807190 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:49.800859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44547 (local address 127.0.97.2:46793)
0924 19:00:49.801831 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:49.801840 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:49.801857 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:00:49.801924 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:49.801930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:49.802008 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:49.802123 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:49.802730 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:49.803666 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:49.805372 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:49.806229 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:49.806367 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:49.806494 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:49.806737 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 19:00:49.806868 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":851,"thread_start_us":758,"threads_started":1}
I20260924 19:00:49.807216 8571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:49.800839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46793 (local address 127.0.97.1:44547)
0924 19:00:49.801274 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:49.801289 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:49.801397 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:49.802149 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:49.802153 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:49.802181 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:49.802530 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:49.802543 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:49.803799 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:49.803804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:49.805225 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:49.805236 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:49.806408 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:49.806414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:49.806587 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:49.806612 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 19:00:49.806714 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 19:00:49.806850 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":228,"threads_started":1}
I20260924 19:00:49.807559 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:49.801772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39087 (local address 127.0.97.3:35721)
0924 19:00:49.801931 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:49.801936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:49.802424 (+ 488us) server_negotiation.cc:407] Connection header received
0924 19:00:49.802620 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:49.802626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:49.802705 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:49.802811 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:00:49.803393 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:49.804099 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:49.805797 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:49.806575 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:49.806727 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:49.806828 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:49.807126 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 19:00:49.807253 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:00:49.807682 8573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:49.801636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35721 (local address 127.0.97.1:39087)
0924 19:00:49.802288 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:49.802312 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:49.802469 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:49.802840 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:49.802845 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:49.802866 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:00:49.803240 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:49.803249 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:49.804226 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:49.804233 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:49.805648 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:49.805659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:49.806807 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:49.806813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:49.806974 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:49.807000 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 19:00:49.807102 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 19:00:49.807220 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":321,"threads_started":1}
I20260924 19:00:49.807947 8278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019636196b2a4b518ce6b9383cba5312" candidate_uuid: "934856ed75d54e2cbc38989df8f509d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" is_pre_election: true
I20260924 19:00:49.808199 8278 raft_consensus.cc:2454] T 019636196b2a4b518ce6b9383cba5312 P 69ea0ce743ff4c8ca80afa6c7da26941 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 934856ed75d54e2cbc38989df8f509d1 in term 0.
I20260924 19:00:49.808717 8069 leader_election.cc:302] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ea0ce743ff4c8ca80afa6c7da26941, 934856ed75d54e2cbc38989df8f509d1; no voters:
I20260924 19:00:49.809161 8498 raft_consensus.cc:2781] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:00:49.809283 8498 raft_consensus.cc:482] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:00:49.809090 8421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019636196b2a4b518ce6b9383cba5312" candidate_uuid: "934856ed75d54e2cbc38989df8f509d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e31a53118f4d45ad946d20cf4952a46b" is_pre_election: true
I20260924 19:00:49.809383 8498 raft_consensus.cc:3037] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:49.809533 8421 raft_consensus.cc:2454] T 019636196b2a4b518ce6b9383cba5312 P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 934856ed75d54e2cbc38989df8f509d1 in term 0.
I20260924 19:00:49.811596 8498 raft_consensus.cc:504] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:49.812052 8498 leader_election.cc:288] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1 [CANDIDATE]: Term 1 election: Requested vote from peers 69ea0ce743ff4c8ca80afa6c7da26941 (127.0.97.2:46793), e31a53118f4d45ad946d20cf4952a46b (127.0.97.3:35721)
I20260924 19:00:49.812469 8278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019636196b2a4b518ce6b9383cba5312" candidate_uuid: "934856ed75d54e2cbc38989df8f509d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ea0ce743ff4c8ca80afa6c7da26941"
I20260924 19:00:49.812537 8421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "019636196b2a4b518ce6b9383cba5312" candidate_uuid: "934856ed75d54e2cbc38989df8f509d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e31a53118f4d45ad946d20cf4952a46b"
I20260924 19:00:49.812700 8278 raft_consensus.cc:3037] T 019636196b2a4b518ce6b9383cba5312 P 69ea0ce743ff4c8ca80afa6c7da26941 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:49.812712 8421 raft_consensus.cc:3037] T 019636196b2a4b518ce6b9383cba5312 P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:49.815608 8278 raft_consensus.cc:2454] T 019636196b2a4b518ce6b9383cba5312 P 69ea0ce743ff4c8ca80afa6c7da26941 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 934856ed75d54e2cbc38989df8f509d1 in term 1.
I20260924 19:00:49.815634 8421 raft_consensus.cc:2454] T 019636196b2a4b518ce6b9383cba5312 P e31a53118f4d45ad946d20cf4952a46b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 934856ed75d54e2cbc38989df8f509d1 in term 1.
I20260924 19:00:49.815965 8069 leader_election.cc:302] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ea0ce743ff4c8ca80afa6c7da26941, 934856ed75d54e2cbc38989df8f509d1; no voters:
I20260924 19:00:49.816237 8498 raft_consensus.cc:2781] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:00:49.816805 8498 raft_consensus.cc:686] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1 [term 1 LEADER]: Becoming Leader. State: Replica: 934856ed75d54e2cbc38989df8f509d1, State: Running, Role: LEADER
I20260924 19:00:49.817196 8498 consensus_queue.cc:237] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:49.820425 7982 catalog_manager.cc:5495] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 934856ed75d54e2cbc38989df8f509d1 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "934856ed75d54e2cbc38989df8f509d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } health_report { overall_health: UNKNOWN } } }
I20260924 19:00:49.847707 8577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:49.841588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48442 (local address 127.0.97.1:40683)
0924 19:00:49.842169 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:49.842173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:49.842192 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:00:49.842274 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:49.842278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:49.842344 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:49.842476 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:00:49.842834 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:49.843581 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:49.845545 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:49.845954 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:49.846045 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:49.846520 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:00:49.847078 (+ 558us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:00:49.847186 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:49.847314 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 19:00:49.847373 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":322,"threads_started":1}
I20260924 19:00:49.851289 8278 raft_consensus.cc:1261] T 019636196b2a4b518ce6b9383cba5312 P 69ea0ce743ff4c8ca80afa6c7da26941 [term 1 FOLLOWER]: Refusing update from remote peer 934856ed75d54e2cbc38989df8f509d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:00:49.851444 8421 raft_consensus.cc:1261] T 019636196b2a4b518ce6b9383cba5312 P e31a53118f4d45ad946d20cf4952a46b [term 1 FOLLOWER]: Refusing update from remote peer 934856ed75d54e2cbc38989df8f509d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:00:49.851994 8574 consensus_queue.cc:1035] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:00:49.852339 8498 consensus_queue.cc:1035] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:00:49.862658 8549 mvcc.cc:205] Tried to move back new op lower bound from 7332972338584301568 to 7332972338455310336. Current Snapshot: MvccSnapshot[applied={T|T < 7332972338584301568}]
I20260924 19:00:49.869614 7980 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:00:49.869858 7980 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:00:49.872862 7980 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a099787801042bcb43901bb50891277: Sending DeleteTablet for 3 replicas of tablet 019636196b2a4b518ce6b9383cba5312
I20260924 19:00:49.873555 8118 tablet_service.cc:1514] Processing DeleteTablet for tablet 019636196b2a4b518ce6b9383cba5312 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:49 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48440
I20260924 19:00:49.873875 8258 tablet_service.cc:1514] Processing DeleteTablet for tablet 019636196b2a4b518ce6b9383cba5312 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:49 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55670
I20260924 19:00:49.874140 8554 tablet_replica.cc:330] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1: stopping tablet replica
I20260924 19:00:49.874341 8554 raft_consensus.cc:2229] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:00:49.874663 7980 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:00:49.874790 8554 raft_consensus.cc:2258] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:49.875052 8553 tablet_replica.cc:330] T 019636196b2a4b518ce6b9383cba5312 P 69ea0ce743ff4c8ca80afa6c7da26941: stopping tablet replica
I20260924 19:00:49.875232 8553 raft_consensus.cc:2229] T 019636196b2a4b518ce6b9383cba5312 P 69ea0ce743ff4c8ca80afa6c7da26941 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 19:00:49.875257 7980 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:00:49.875427 8553 raft_consensus.cc:2258] T 019636196b2a4b518ce6b9383cba5312 P 69ea0ce743ff4c8ca80afa6c7da26941 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:49.876255 8396 tablet_service.cc:1514] Processing DeleteTablet for tablet 019636196b2a4b518ce6b9383cba5312 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:49 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52416
I20260924 19:00:49.876871 8555 tablet_replica.cc:330] T 019636196b2a4b518ce6b9383cba5312 P e31a53118f4d45ad946d20cf4952a46b: stopping tablet replica
I20260924 19:00:49.876966 8554 ts_tablet_manager.cc:1905] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:49.877039 8555 raft_consensus.cc:2229] T 019636196b2a4b518ce6b9383cba5312 P e31a53118f4d45ad946d20cf4952a46b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:00:49.877247 8555 raft_consensus.cc:2258] T 019636196b2a4b518ce6b9383cba5312 P e31a53118f4d45ad946d20cf4952a46b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:49.878667 8555 ts_tablet_manager.cc:1905] T 019636196b2a4b518ce6b9383cba5312 P e31a53118f4d45ad946d20cf4952a46b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:49.882552 8258 tablet_service.cc:1467] Processing CreateTablet for tablet 600389571ba6470894d26952e542116e (DEFAULT_TABLE table=test-table [id=daf043a3ad4648ccbcafe4834dc4596d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:49.882946 8258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 600389571ba6470894d26952e542116e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:49.883729 8554 ts_tablet_manager.cc:1918] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:00:49.883883 8554 log.cc:1196] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-0/wal/wals/019636196b2a4b518ce6b9383cba5312
I20260924 19:00:49.884346 8554 ts_tablet_manager.cc:1939] T 019636196b2a4b518ce6b9383cba5312 P 934856ed75d54e2cbc38989df8f509d1: Deleting consensus metadata
I20260924 19:00:49.885406 8555 ts_tablet_manager.cc:1918] T 019636196b2a4b518ce6b9383cba5312 P e31a53118f4d45ad946d20cf4952a46b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:00:49.885632 8555 log.cc:1196] T 019636196b2a4b518ce6b9383cba5312 P e31a53118f4d45ad946d20cf4952a46b: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-2/wal/wals/019636196b2a4b518ce6b9383cba5312
I20260924 19:00:49.885928 8553 ts_tablet_manager.cc:1905] T 019636196b2a4b518ce6b9383cba5312 P 69ea0ce743ff4c8ca80afa6c7da26941: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:49.886176 7966 catalog_manager.cc:4841] TS 934856ed75d54e2cbc38989df8f509d1 (127.0.97.1:40683): tablet 019636196b2a4b518ce6b9383cba5312 (table test-table [id=6ab8bfa944af4d0f92b156a013951abc]) successfully deleted
I20260924 19:00:49.886077 8396 tablet_service.cc:1467] Processing CreateTablet for tablet 600389571ba6470894d26952e542116e (DEFAULT_TABLE table=test-table [id=daf043a3ad4648ccbcafe4834dc4596d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:49.886564 8396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 600389571ba6470894d26952e542116e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:49.889869 8118 tablet_service.cc:1467] Processing CreateTablet for tablet 600389571ba6470894d26952e542116e (DEFAULT_TABLE table=test-table [id=daf043a3ad4648ccbcafe4834dc4596d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:00:49.890245 8118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 600389571ba6470894d26952e542116e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:49.890712 8493 tablet_bootstrap.cc:492] T 600389571ba6470894d26952e542116e P 69ea0ce743ff4c8ca80afa6c7da26941: Bootstrap starting.
I20260924 19:00:49.892304 8493 tablet_bootstrap.cc:654] T 600389571ba6470894d26952e542116e P 69ea0ce743ff4c8ca80afa6c7da26941: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:49.896157 8492 tablet_bootstrap.cc:492] T 600389571ba6470894d26952e542116e P e31a53118f4d45ad946d20cf4952a46b: Bootstrap starting.
I20260924 19:00:49.897782 8492 tablet_bootstrap.cc:654] T 600389571ba6470894d26952e542116e P e31a53118f4d45ad946d20cf4952a46b: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:49.898123 8555 ts_tablet_manager.cc:1939] T 019636196b2a4b518ce6b9383cba5312 P e31a53118f4d45ad946d20cf4952a46b: Deleting consensus metadata
I20260924 19:00:49.899471 7966 catalog_manager.cc:4841] TS e31a53118f4d45ad946d20cf4952a46b (127.0.97.3:35721): tablet 019636196b2a4b518ce6b9383cba5312 (table test-table [id=6ab8bfa944af4d0f92b156a013951abc]) successfully deleted
I20260924 19:00:49.901479 8493 tablet_bootstrap.cc:492] T 600389571ba6470894d26952e542116e P 69ea0ce743ff4c8ca80afa6c7da26941: No bootstrap required, opened a new log
I20260924 19:00:49.901703 8493 ts_tablet_manager.cc:1397] T 600389571ba6470894d26952e542116e P 69ea0ce743ff4c8ca80afa6c7da26941: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260924 19:00:49.902379 8491 tablet_bootstrap.cc:492] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1: Bootstrap starting.
I20260924 19:00:49.902479 8493 raft_consensus.cc:348] T 600389571ba6470894d26952e542116e P 69ea0ce743ff4c8ca80afa6c7da26941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:49.902695 8493 raft_consensus.cc:374] T 600389571ba6470894d26952e542116e P 69ea0ce743ff4c8ca80afa6c7da26941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:49.902768 8493 raft_consensus.cc:729] T 600389571ba6470894d26952e542116e P 69ea0ce743ff4c8ca80afa6c7da26941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ea0ce743ff4c8ca80afa6c7da26941, State: Initialized, Role: FOLLOWER
I20260924 19:00:49.903018 8493 consensus_queue.cc:260] T 600389571ba6470894d26952e542116e P 69ea0ce743ff4c8ca80afa6c7da26941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:49.903682 8553 ts_tablet_manager.cc:1918] T 019636196b2a4b518ce6b9383cba5312 P 69ea0ce743ff4c8ca80afa6c7da26941: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:00:49.903826 8553 log.cc:1196] T 019636196b2a4b518ce6b9383cba5312 P 69ea0ce743ff4c8ca80afa6c7da26941: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-1/wal/wals/019636196b2a4b518ce6b9383cba5312
I20260924 19:00:49.904067 8491 tablet_bootstrap.cc:654] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:49.904275 8553 ts_tablet_manager.cc:1939] T 019636196b2a4b518ce6b9383cba5312 P 69ea0ce743ff4c8ca80afa6c7da26941: Deleting consensus metadata
I20260924 19:00:49.905252 7966 catalog_manager.cc:4841] TS 69ea0ce743ff4c8ca80afa6c7da26941 (127.0.97.2:46793): tablet 019636196b2a4b518ce6b9383cba5312 (table test-table [id=6ab8bfa944af4d0f92b156a013951abc]) successfully deleted
I20260924 19:00:49.906919 8493 ts_tablet_manager.cc:1428] T 600389571ba6470894d26952e542116e P 69ea0ce743ff4c8ca80afa6c7da26941: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260924 19:00:49.910851 8491 tablet_bootstrap.cc:492] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1: No bootstrap required, opened a new log
I20260924 19:00:49.911036 8491 ts_tablet_manager.cc:1397] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260924 19:00:49.911216 8492 tablet_bootstrap.cc:492] T 600389571ba6470894d26952e542116e P e31a53118f4d45ad946d20cf4952a46b: No bootstrap required, opened a new log
I20260924 19:00:49.911386 8492 ts_tablet_manager.cc:1397] T 600389571ba6470894d26952e542116e P e31a53118f4d45ad946d20cf4952a46b: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20260924 19:00:49.911796 8491 raft_consensus.cc:348] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:49.912016 8491 raft_consensus.cc:374] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:49.912111 8491 raft_consensus.cc:729] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 934856ed75d54e2cbc38989df8f509d1, State: Initialized, Role: FOLLOWER
I20260924 19:00:49.912139 8492 raft_consensus.cc:348] T 600389571ba6470894d26952e542116e P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:49.912384 8492 raft_consensus.cc:374] T 600389571ba6470894d26952e542116e P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:49.912460 8492 raft_consensus.cc:729] T 600389571ba6470894d26952e542116e P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e31a53118f4d45ad946d20cf4952a46b, State: Initialized, Role: FOLLOWER
I20260924 19:00:49.912369 8491 consensus_queue.cc:260] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:49.912734 8492 consensus_queue.cc:260] T 600389571ba6470894d26952e542116e P e31a53118f4d45ad946d20cf4952a46b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:49.912887 8491 ts_tablet_manager.cc:1428] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 19:00:49.913232 8492 ts_tablet_manager.cc:1428] T 600389571ba6470894d26952e542116e P e31a53118f4d45ad946d20cf4952a46b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:00:50.158409 8574 raft_consensus.cc:482] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:00:50.158700 8574 raft_consensus.cc:504] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:50.159402 8574 leader_election.cc:288] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69ea0ce743ff4c8ca80afa6c7da26941 (127.0.97.2:46793), e31a53118f4d45ad946d20cf4952a46b (127.0.97.3:35721)
I20260924 19:00:50.160156 8421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "600389571ba6470894d26952e542116e" candidate_uuid: "934856ed75d54e2cbc38989df8f509d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e31a53118f4d45ad946d20cf4952a46b" is_pre_election: true
I20260924 19:00:50.160182 8278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "600389571ba6470894d26952e542116e" candidate_uuid: "934856ed75d54e2cbc38989df8f509d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" is_pre_election: true
I20260924 19:00:50.160408 8421 raft_consensus.cc:2454] T 600389571ba6470894d26952e542116e P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 934856ed75d54e2cbc38989df8f509d1 in term 0.
I20260924 19:00:50.160475 8278 raft_consensus.cc:2454] T 600389571ba6470894d26952e542116e P 69ea0ce743ff4c8ca80afa6c7da26941 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 934856ed75d54e2cbc38989df8f509d1 in term 0.
I20260924 19:00:50.160902 8069 leader_election.cc:302] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 934856ed75d54e2cbc38989df8f509d1, e31a53118f4d45ad946d20cf4952a46b; no voters:
I20260924 19:00:50.161206 8574 raft_consensus.cc:2781] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:00:50.161324 8574 raft_consensus.cc:482] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:00:50.161461 8574 raft_consensus.cc:3037] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:50.163712 8574 raft_consensus.cc:504] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:50.164265 8574 leader_election.cc:288] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1 [CANDIDATE]: Term 1 election: Requested vote from peers 69ea0ce743ff4c8ca80afa6c7da26941 (127.0.97.2:46793), e31a53118f4d45ad946d20cf4952a46b (127.0.97.3:35721)
I20260924 19:00:50.164790 8278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "600389571ba6470894d26952e542116e" candidate_uuid: "934856ed75d54e2cbc38989df8f509d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ea0ce743ff4c8ca80afa6c7da26941"
I20260924 19:00:50.165007 8278 raft_consensus.cc:3037] T 600389571ba6470894d26952e542116e P 69ea0ce743ff4c8ca80afa6c7da26941 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:50.167238 8278 raft_consensus.cc:2454] T 600389571ba6470894d26952e542116e P 69ea0ce743ff4c8ca80afa6c7da26941 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 934856ed75d54e2cbc38989df8f509d1 in term 1.
I20260924 19:00:50.167696 8069 leader_election.cc:302] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ea0ce743ff4c8ca80afa6c7da26941, 934856ed75d54e2cbc38989df8f509d1; no voters:
I20260924 19:00:50.167675 8421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "600389571ba6470894d26952e542116e" candidate_uuid: "934856ed75d54e2cbc38989df8f509d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e31a53118f4d45ad946d20cf4952a46b"
I20260924 19:00:50.167879 8421 raft_consensus.cc:3037] T 600389571ba6470894d26952e542116e P e31a53118f4d45ad946d20cf4952a46b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:50.167982 8574 raft_consensus.cc:2781] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:00:50.168161 8574 raft_consensus.cc:686] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1 [term 1 LEADER]: Becoming Leader. State: Replica: 934856ed75d54e2cbc38989df8f509d1, State: Running, Role: LEADER
I20260924 19:00:50.168426 8574 consensus_queue.cc:237] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } }
I20260924 19:00:50.170715 8421 raft_consensus.cc:2454] T 600389571ba6470894d26952e542116e P e31a53118f4d45ad946d20cf4952a46b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 934856ed75d54e2cbc38989df8f509d1 in term 1.
I20260924 19:00:50.171511 7982 catalog_manager.cc:5495] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 934856ed75d54e2cbc38989df8f509d1 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "934856ed75d54e2cbc38989df8f509d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934856ed75d54e2cbc38989df8f509d1" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40683 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 } health_report { overall_health: UNKNOWN } } }
I20260924 19:00:50.220269 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:50.214006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54679 (local address 127.0.97.3:35721)
0924 19:00:50.214197 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:50.214204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:50.214940 (+ 736us) server_negotiation.cc:407] Connection header received
0924 19:00:50.215089 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:50.215095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:50.215179 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:50.215284 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:00:50.215740 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:50.216834 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:50.218351 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:50.218971 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:50.219110 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:50.219523 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:00:50.219641 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:00:50.219747 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:50.219886 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 19:00:50.219960 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 19:00:50.220402 8591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:50.214090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35721 (local address 127.0.97.1:54679)
0924 19:00:50.214832 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:50.214850 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:50.214962 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:50.215302 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:50.215305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:50.215314 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:00:50.215596 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:50.215602 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:50.216966 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:50.216970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:50.218202 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:50.218214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:50.219138 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:50.219142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:50.219291 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:50.219343 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:00:50.219743 (+ 400us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:00:50.219758 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 19:00:50.219855 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 19:00:50.219990 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":413,"threads_started":1}
I20260924 19:00:50.228544 8421 raft_consensus.cc:1261] T 600389571ba6470894d26952e542116e P e31a53118f4d45ad946d20cf4952a46b [term 1 FOLLOWER]: Refusing update from remote peer 934856ed75d54e2cbc38989df8f509d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:00:50.228626 8278 raft_consensus.cc:1261] T 600389571ba6470894d26952e542116e P 69ea0ce743ff4c8ca80afa6c7da26941 [term 1 FOLLOWER]: Refusing update from remote peer 934856ed75d54e2cbc38989df8f509d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:00:50.229126 8574 consensus_queue.cc:1035] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e31a53118f4d45ad946d20cf4952a46b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:00:50.229431 8498 consensus_queue.cc:1035] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ea0ce743ff4c8ca80afa6c7da26941" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:00:50.234169 8538 mvcc.cc:205] Tried to move back new op lower bound from 7332972340130611200 to 7332972339892858880. Current Snapshot: MvccSnapshot[applied={T|T < 7332972340130611200}]
I20260924 19:00:50.257822 8577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:50.250290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60781 (local address 127.0.97.1:40683)
0924 19:00:50.250482 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:50.250491 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:50.250670 (+ 179us) server_negotiation.cc:407] Connection header received
0924 19:00:50.251158 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:50.251164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:50.251256 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:50.251355 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:00:50.252157 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:50.253276 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:50.255212 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:50.255929 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:50.256058 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:50.257114 (+ 1056us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:00:50.257229 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:00:50.257315 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:50.257415 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 19:00:50.257486 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 19:00:50.257894 8595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:50.249641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40683 (local address 127.0.97.3:60781)
0924 19:00:50.250576 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:50.250597 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:50.250775 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:50.251693 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:50.251701 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:50.251717 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:00:50.251966 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:50.251977 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:50.253451 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:50.253456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:50.255036 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:50.255051 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:50.256760 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:50.256765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:50.256914 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:50.256962 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:00:50.257338 (+ 376us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:00:50.257345 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 19:00:50.257398 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 19:00:50.257491 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"thread_start_us":5393,"threads_started":1}
I20260924 19:00:50.302174 7982 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:00:50.302416 7982 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:00:50.305554 7982 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a099787801042bcb43901bb50891277: Sending DeleteTablet for 3 replicas of tablet 600389571ba6470894d26952e542116e
I20260924 19:00:50.306429 8258 tablet_service.cc:1514] Processing DeleteTablet for tablet 600389571ba6470894d26952e542116e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:50 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55670
I20260924 19:00:50.306627 8118 tablet_service.cc:1514] Processing DeleteTablet for tablet 600389571ba6470894d26952e542116e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:50 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48440
I20260924 19:00:50.306736 8553 tablet_replica.cc:330] T 600389571ba6470894d26952e542116e P 69ea0ce743ff4c8ca80afa6c7da26941: stopping tablet replica
I20260924 19:00:50.306917 8553 raft_consensus.cc:2229] T 600389571ba6470894d26952e542116e P 69ea0ce743ff4c8ca80afa6c7da26941 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:00:50.307003 8554 tablet_replica.cc:330] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1: stopping tablet replica
I20260924 19:00:50.307114 8553 raft_consensus.cc:2258] T 600389571ba6470894d26952e542116e P 69ea0ce743ff4c8ca80afa6c7da26941 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:50.307159 8554 raft_consensus.cc:2229] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:00:50.307422 8396 tablet_service.cc:1514] Processing DeleteTablet for tablet 600389571ba6470894d26952e542116e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:00:50 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52416
I20260924 19:00:50.307467 8554 raft_consensus.cc:2258] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:50.307884 8555 tablet_replica.cc:330] T 600389571ba6470894d26952e542116e P e31a53118f4d45ad946d20cf4952a46b: stopping tablet replica
I20260924 19:00:50.308091 8555 raft_consensus.cc:2229] T 600389571ba6470894d26952e542116e P e31a53118f4d45ad946d20cf4952a46b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:00:50.308297 8555 raft_consensus.cc:2258] T 600389571ba6470894d26952e542116e P e31a53118f4d45ad946d20cf4952a46b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:00:50.308506 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 8053
I20260924 19:00:50.309809 8553 ts_tablet_manager.cc:1905] T 600389571ba6470894d26952e542116e P 69ea0ce743ff4c8ca80afa6c7da26941: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:50.310931 8554 ts_tablet_manager.cc:1905] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:50.316365 8554 ts_tablet_manager.cc:1918] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:00:50.316463 8553 ts_tablet_manager.cc:1918] T 600389571ba6470894d26952e542116e P 69ea0ce743ff4c8ca80afa6c7da26941: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:00:50.316512 8554 log.cc:1196] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-0/wal/wals/600389571ba6470894d26952e542116e
I20260924 19:00:50.316604 8553 log.cc:1196] T 600389571ba6470894d26952e542116e P 69ea0ce743ff4c8ca80afa6c7da26941: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-1/wal/wals/600389571ba6470894d26952e542116e
I20260924 19:00:50.316920 8554 ts_tablet_manager.cc:1939] T 600389571ba6470894d26952e542116e P 934856ed75d54e2cbc38989df8f509d1: Deleting consensus metadata
I20260924 19:00:50.317688 8553 ts_tablet_manager.cc:1939] T 600389571ba6470894d26952e542116e P 69ea0ce743ff4c8ca80afa6c7da26941: Deleting consensus metadata
I20260924 19:00:50.317795 8555 ts_tablet_manager.cc:1905] T 600389571ba6470894d26952e542116e P e31a53118f4d45ad946d20cf4952a46b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:00:50.318590 7966 catalog_manager.cc:4841] TS 69ea0ce743ff4c8ca80afa6c7da26941 (127.0.97.2:46793): tablet 600389571ba6470894d26952e542116e (table test-table [id=daf043a3ad4648ccbcafe4834dc4596d]) successfully deleted
I20260924 19:00:50.318819 7966 catalog_manager.cc:4841] TS 934856ed75d54e2cbc38989df8f509d1 (127.0.97.1:40683): tablet 600389571ba6470894d26952e542116e (table test-table [id=daf043a3ad4648ccbcafe4834dc4596d]) successfully deleted
I20260924 19:00:50.323308 8555 ts_tablet_manager.cc:1918] T 600389571ba6470894d26952e542116e P e31a53118f4d45ad946d20cf4952a46b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:00:50.323444 8555 log.cc:1196] T 600389571ba6470894d26952e542116e P e31a53118f4d45ad946d20cf4952a46b: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276386554690-388-0/minicluster-data/ts-2/wal/wals/600389571ba6470894d26952e542116e
I20260924 19:00:50.323815 8555 ts_tablet_manager.cc:1939] T 600389571ba6470894d26952e542116e P e31a53118f4d45ad946d20cf4952a46b: Deleting consensus metadata
Sep 24 19:00:50 dist-test-slave-9fn1 krb5kdc[7937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260924 19:00:50.324693 7966 catalog_manager.cc:4841] TS e31a53118f4d45ad946d20cf4952a46b (127.0.97.3:35721): tablet 600389571ba6470894d26952e542116e (table test-table [id=daf043a3ad4648ccbcafe4834dc4596d]) successfully deleted
I20260924 19:00:50.331132 8577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:50.309411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48458 (local address 127.0.97.1:40683)
0924 19:00:50.309717 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:50.309724 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:50.310057 (+ 333us) server_negotiation.cc:407] Connection header received
0924 19:00:50.310344 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:50.310350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:50.310423 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:50.310512 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:50.312323 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:50.313374 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:50.315225 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:50.315879 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:50.315987 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:50.325103 (+ 9116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:50.325115 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:50.325132 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:50.325165 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:50.328317 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:50.329042 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:50.329048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:50.329050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:50.329164 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:50.329599 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:50.329602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:50.329604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:50.329932 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:50.330140 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:50.330479 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 19:00:50.330650 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260924 19:00:50.331749 8184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:50.468292 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 8053
I20260924 19:00:50.490221 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 8193
Sep 24 19:00:50 dist-test-slave-9fn1 krb5kdc[7937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260924 19:00:50.504149 8599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:50.490943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55672 (local address 127.0.97.2:46793)
0924 19:00:50.491424 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:50.491428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:50.491444 (+ 16us) server_negotiation.cc:407] Connection header received
0924 19:00:50.491501 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:50.491506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:50.491562 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:50.491654 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:50.492781 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:50.493482 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:50.494624 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:50.495044 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:50.495133 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:50.498936 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:50.498948 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:50.498964 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:50.498994 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:50.501974 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:50.502511 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:50.502517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:50.502519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:50.502634 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:50.502916 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:50.502918 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:50.502920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:50.503236 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:50.503434 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:50.503624 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 19:00:50.503775 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":262,"threads_started":1}
I20260924 19:00:50.504760 8323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:50.601974 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 8193
I20260924 19:00:50.620043 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 8332
Sep 24 19:00:50 dist-test-slave-9fn1 krb5kdc[7937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260924 19:00:50.634332 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:50.620791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52442 (local address 127.0.97.3:35721)
0924 19:00:50.620963 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:50.620970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:50.621013 (+ 43us) server_negotiation.cc:407] Connection header received
0924 19:00:50.621235 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:50.621240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:50.621299 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:50.621415 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:50.622674 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:50.623409 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:50.624627 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:50.625014 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:50.625147 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:50.629252 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:50.629266 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:50.629281 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:50.629315 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:50.631801 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:50.632353 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:50.632360 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:50.632362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:50.632491 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:50.632938 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:50.632941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:50.632943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:50.633267 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:50.633477 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:50.633895 (+ 418us) server_negotiation.cc:299] Negotiation successful
0924 19:00:50.634070 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 19:00:50.634970 8462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:50.774204 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 8332
I20260924 19:00:50.798586 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 7950
I20260924 19:00:50.799711 8022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:50.911100 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 7950
2026-09-24T19:00:50Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5904 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:00:51 dist-test-slave-9fn1 krb5kdc[8606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:00:51 dist-test-slave-9fn1 krb5kdc[8606](info): set up 2 sockets
Sep 24 19:00:51 dist-test-slave-9fn1 krb5kdc[8606](info): commencing operation
krb5kdc: starting...
W20260924 19:00:51.998416 388 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:00:52 dist-test-slave-9fn1 krb5kdc[8606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:00:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:00:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 19:00:52.208611 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40373
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:52.515836 8619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:52.516153 8619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:52.516307 8619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:52.523203 8619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:00:52.523345 8619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:52.523437 8619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:52.523514 8619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:00:52.523581 8619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:00:52.532837 8619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:52.534085 8619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:52.535349 8619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:52.546793 8624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:52.546878 8625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:52.546926 8627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:52.548060 8619 server_base.cc:1034] running on GCE node
I20260924 19:00:52.548838 8619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:52.550213 8619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:52.551419 8619 hybrid_clock.cc:648] HybridClock initialized: now 1790276452551359 us; error 53 us; skew 500 ppm
I20260924 19:00:52.551709 8619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T19:00:52Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1712 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:00:52 dist-test-slave-9fn1 krb5kdc[8633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:00:52 dist-test-slave-9fn1 krb5kdc[8633](info): set up 2 sockets
Sep 24 19:00:52 dist-test-slave-9fn1 krb5kdc[8633](info): commencing operation
krb5kdc: starting...
W20260924 19:00:53.714397 388 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.006s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:00:53 dist-test-slave-9fn1 krb5kdc[8633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:00:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:00:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 19:00:53.925022 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42585
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:54.238972 8646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:54.239306 8646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:54.239459 8646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:54.246481 8646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:00:54.246613 8646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:54.246712 8646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:54.246817 8646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:00:54.246903 8646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:00:54.256402 8646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:54.257741 8646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:54.259029 8646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:54.270701 8651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:54.270710 8652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:54.270759 8654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:54.271548 8646 server_base.cc:1034] running on GCE node
I20260924 19:00:54.272327 8646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:54.273746 8646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:54.274979 8646 hybrid_clock.cc:648] HybridClock initialized: now 1790276454274925 us; error 61 us; skew 500 ppm
I20260924 19:00:54.275322 8646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:54 dist-test-slave-9fn1 krb5kdc[8633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:54.278472 8646 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:00:54.279824 8646 webserver.cc:460] Webserver started at http://127.0.97.62:35919/ using document root <none> and password file <none>
I20260924 19:00:54.280445 8646 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:54.280540 8646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:54.280790 8646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:54.283592 8646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "c69e520e097c4d4aa8cc9c6a254b83ae"
format_stamp: "Formatted at 2026-09-24 19:00:54 on dist-test-slave-9fn1"
server_key: "c4fe0a4de0b503f1f5c27208e2307c0b"
server_key_iv: "0b353e1a5ecfc5f5e0db3a38890b4acc"
server_key_version: "encryptionkey@0"
I20260924 19:00:54.284291 8646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "c69e520e097c4d4aa8cc9c6a254b83ae"
format_stamp: "Formatted at 2026-09-24 19:00:54 on dist-test-slave-9fn1"
server_key: "c4fe0a4de0b503f1f5c27208e2307c0b"
server_key_iv: "0b353e1a5ecfc5f5e0db3a38890b4acc"
server_key_version: "encryptionkey@0"
I20260924 19:00:54.288996 8646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260924 19:00:54.292213 8661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:54.293422 8646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 19:00:54.293603 8646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "c69e520e097c4d4aa8cc9c6a254b83ae"
format_stamp: "Formatted at 2026-09-24 19:00:54 on dist-test-slave-9fn1"
server_key: "c4fe0a4de0b503f1f5c27208e2307c0b"
server_key_iv: "0b353e1a5ecfc5f5e0db3a38890b4acc"
server_key_version: "encryptionkey@0"
I20260924 19:00:54.293751 8646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:54.317550 8646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:54.321233 8646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:54.321450 8646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:54.338940 8646 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42585
I20260924 19:00:54.338977 8712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42585 every 8 connection(s)
I20260924 19:00:54.340286 8646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:00:54.343293 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 8646
I20260924 19:00:54.343482 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:00:54.344287 388 external_mini_cluster.cc:1428] Setting key eed42067ca9f29dbdfe85822c81a5621
I20260924 19:00:54.344306 8713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:00:54.358512 8713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae: Bootstrap starting.
Sep 24 19:00:54 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:54.361780 8713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae: Neither blocks nor log segments found. Creating new log.
I20260924 19:00:54.362787 8713 log.cc:824] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:54.365348 8713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae: No bootstrap required, opened a new log
I20260924 19:00:54.366662 8716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:54.346396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60434 (local address 127.0.97.62:42585)
0924 19:00:54.347150 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:54.347161 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:54.347204 (+ 43us) server_negotiation.cc:407] Connection header received
0924 19:00:54.348928 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:54.348947 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:54.349326 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:54.349732 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:54.351460 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:54.352757 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:54.354528 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:54.355241 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:54.355379 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:54.359838 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:54.359845 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:54.359859 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:54.359893 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:54.362864 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:54.363611 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:54.363617 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:54.363623 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:54.363782 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:54.364174 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:54.364183 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:54.364185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:54.364629 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:54.364905 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:54.365311 (+ 406us) server_negotiation.cc:299] Negotiation successful
0924 19:00:54.365766 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":330,"threads_started":1}
I20260924 19:00:54.372527 8713 raft_consensus.cc:348] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69e520e097c4d4aa8cc9c6a254b83ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42585 } }
I20260924 19:00:54.372874 8713 raft_consensus.cc:374] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:00:54.372977 8713 raft_consensus.cc:729] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c69e520e097c4d4aa8cc9c6a254b83ae, State: Initialized, Role: FOLLOWER
I20260924 19:00:54.373486 8713 consensus_queue.cc:260] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69e520e097c4d4aa8cc9c6a254b83ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42585 } }
I20260924 19:00:54.373772 8713 raft_consensus.cc:388] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:00:54.373908 8713 raft_consensus.cc:482] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:00:54.374089 8713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:00:54.376092 8713 raft_consensus.cc:504] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69e520e097c4d4aa8cc9c6a254b83ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42585 } }
I20260924 19:00:54.376564 8713 leader_election.cc:302] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c69e520e097c4d4aa8cc9c6a254b83ae; no voters:
I20260924 19:00:54.377087 8713 leader_election.cc:288] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:00:54.377229 8718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:00:54.377908 8718 raft_consensus.cc:686] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [term 1 LEADER]: Becoming Leader. State: Replica: c69e520e097c4d4aa8cc9c6a254b83ae, State: Running, Role: LEADER
I20260924 19:00:54.378477 8718 consensus_queue.cc:237] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69e520e097c4d4aa8cc9c6a254b83ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42585 } }
I20260924 19:00:54.379048 8713 sys_catalog.cc:564] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:00:54.381119 8720 sys_catalog.cc:455] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader c69e520e097c4d4aa8cc9c6a254b83ae. Latest consensus state: current_term: 1 leader_uuid: "c69e520e097c4d4aa8cc9c6a254b83ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69e520e097c4d4aa8cc9c6a254b83ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42585 } } }
I20260924 19:00:54.381686 8719 sys_catalog.cc:455] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c69e520e097c4d4aa8cc9c6a254b83ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69e520e097c4d4aa8cc9c6a254b83ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42585 } } }
I20260924 19:00:54.381989 8719 sys_catalog.cc:458] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:54.381480 8720 sys_catalog.cc:458] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:54.383829 8724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:00:54.388696 8724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:00:54.398278 8724 catalog_manager.cc:1347] Generated new cluster ID: d86f4bf8125149c48e269c4a7d1174bc
I20260924 19:00:54.398379 8724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:00:54.419315 8724 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:00:54.420601 8724 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:00:54.431397 8724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae: Generated new TSK 0
I20260924 19:00:54.432245 8724 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 19:00:54.490588 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42585
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:54.814112 8739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:54.814370 8739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:54.814553 8739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:54.821519 8739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:54.821640 8739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:54.821846 8739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:00:54.831223 8739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:54.832443 8739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:54.833716 8739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:54.846491 8744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:54.846553 8745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:54.847206 8747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:54.847508 8739 server_base.cc:1034] running on GCE node
I20260924 19:00:54.848116 8739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:54.849007 8739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:54.850235 8739 hybrid_clock.cc:648] HybridClock initialized: now 1790276454850178 us; error 51 us; skew 500 ppm
I20260924 19:00:54.850554 8739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:54 dist-test-slave-9fn1 krb5kdc[8633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:54.853669 8739 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 19:00:54.855046 8739 webserver.cc:460] Webserver started at http://127.0.97.1:35215/ using document root <none> and password file <none>
I20260924 19:00:54.855717 8739 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:54.855860 8739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:54.856158 8739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:54.859052 8739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/data/instance:
uuid: "2f0857b42806474da2808f93862239d2"
format_stamp: "Formatted at 2026-09-24 19:00:54 on dist-test-slave-9fn1"
server_key: "a6a0cac700197806b386c36a39d541b4"
server_key_iv: "9624e6505572f1c474a47e43486741a3"
server_key_version: "encryptionkey@0"
I20260924 19:00:54.859822 8739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2f0857b42806474da2808f93862239d2"
format_stamp: "Formatted at 2026-09-24 19:00:54 on dist-test-slave-9fn1"
server_key: "a6a0cac700197806b386c36a39d541b4"
server_key_iv: "9624e6505572f1c474a47e43486741a3"
server_key_version: "encryptionkey@0"
I20260924 19:00:54.864786 8739 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:00:54.868304 8754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:54.869678 8739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:00:54.869853 8739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "2f0857b42806474da2808f93862239d2"
format_stamp: "Formatted at 2026-09-24 19:00:54 on dist-test-slave-9fn1"
server_key: "a6a0cac700197806b386c36a39d541b4"
server_key_iv: "9624e6505572f1c474a47e43486741a3"
server_key_version: "encryptionkey@0"
I20260924 19:00:54.870049 8739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:54.885625 8739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:54.889942 8739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:54.890233 8739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:54.891222 8739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:54.892593 8739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:54.892696 8739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:54.892853 8739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:54.892928 8739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:54.925693 8739 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46465
I20260924 19:00:54.925782 8866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46465 every 8 connection(s)
I20260924 19:00:54.926971 8739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:00:54.932536 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 8739
I20260924 19:00:54.932749 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/wal/instance
I20260924 19:00:54.933605 388 external_mini_cluster.cc:1428] Setting key 8c8ae0ed2a33522c99ace94013ff6b9e
Sep 24 19:00:54 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:54.948962 8869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:54.929485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51115 (local address 127.0.97.62:42585)
0924 19:00:54.929977 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:54.929982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:54.930879 (+ 897us) server_negotiation.cc:407] Connection header received
0924 19:00:54.932692 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:54.932697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:54.932788 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:54.932915 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:54.934815 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:54.935824 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:54.938354 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:54.938732 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:54.938822 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:54.942911 (+ 4089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:54.942921 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:54.942929 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:54.942972 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:54.945418 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:54.946427 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:54.946435 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:54.946441 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:54.946594 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:54.947035 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:54.947043 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:54.947049 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:54.947424 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:54.947585 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:54.948241 (+ 656us) server_negotiation.cc:299] Negotiation successful
0924 19:00:54.948484 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":235,"threads_started":1}
I20260924 19:00:54.949620 8870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:54.929987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.1:51115)
0924 19:00:54.930707 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:54.930745 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:54.932418 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:54.933205 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:54.933217 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:54.934022 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:54.934653 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:54.934671 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:54.935980 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:54.935985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:54.938213 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:54.938232 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:54.938954 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:54.938962 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:54.939287 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:54.939987 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:54.940020 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:54.942624 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:54.945614 (+ 2990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:54.945625 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:54.945644 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:54.946285 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:54.946716 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:54.946720 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:54.946722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:54.946910 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:54.947542 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:54.947547 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:54.947858 (+ 311us) client_negotiation.cc:769] Sending connection context
0924 19:00:54.948075 (+ 217us) client_negotiation.cc:240] Negotiation successful
0924 19:00:54.948448 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":313,"threads_started":1}
I20260924 19:00:54.951198 8867 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42585
I20260924 19:00:54.951510 8867 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:54.952392 8867 heartbeater.cc:511] Master 127.0.97.62:42585 requested a full tablet report, sending...
I20260924 19:00:54.954496 8678 ts_manager.cc:194] Registered new tserver with Master: 2f0857b42806474da2808f93862239d2 (127.0.97.1:46465)
I20260924 19:00:54.955904 8678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:51115
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 19:00:54.982533 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42585
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:55.294180 8873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:55.294435 8873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:55.294533 8873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:55.301760 8873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:55.301867 8873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:55.301985 8873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:00:55.311474 8873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:55.312649 8873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:55.313973 8873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:55.327134 8881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:55.327196 8878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:55.327175 8879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:55.328384 8873 server_base.cc:1034] running on GCE node
I20260924 19:00:55.328999 8873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:55.330056 8873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:55.331280 8873 hybrid_clock.cc:648] HybridClock initialized: now 1790276455331233 us; error 42 us; skew 500 ppm
I20260924 19:00:55.331595 8873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:55 dist-test-slave-9fn1 krb5kdc[8633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:55.338070 8873 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 19:00:55.339847 8873 webserver.cc:460] Webserver started at http://127.0.97.2:42443/ using document root <none> and password file <none>
I20260924 19:00:55.340716 8873 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:55.340857 8873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:55.341219 8873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:55.345137 8873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/data/instance:
uuid: "6dc95019870b46e2a668db848fa2e472"
format_stamp: "Formatted at 2026-09-24 19:00:55 on dist-test-slave-9fn1"
server_key: "3669afa14a0934578db0675c4f8f5840"
server_key_iv: "5475afeba934c7be1737ed14a00e082b"
server_key_version: "encryptionkey@0"
I20260924 19:00:55.346125 8873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6dc95019870b46e2a668db848fa2e472"
format_stamp: "Formatted at 2026-09-24 19:00:55 on dist-test-slave-9fn1"
server_key: "3669afa14a0934578db0675c4f8f5840"
server_key_iv: "5475afeba934c7be1737ed14a00e082b"
server_key_version: "encryptionkey@0"
I20260924 19:00:55.352402 8873 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 19:00:55.356611 8888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:55.358093 8873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:00:55.358233 8873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "6dc95019870b46e2a668db848fa2e472"
format_stamp: "Formatted at 2026-09-24 19:00:55 on dist-test-slave-9fn1"
server_key: "3669afa14a0934578db0675c4f8f5840"
server_key_iv: "5475afeba934c7be1737ed14a00e082b"
server_key_version: "encryptionkey@0"
I20260924 19:00:55.358436 8873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:55.379340 8873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:55.383092 8873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:55.383317 8873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:55.384261 8873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:55.385614 8873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:55.385694 8873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:55.385772 8873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:55.385818 8873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:55.418231 8873 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42357
I20260924 19:00:55.418286 9000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42357 every 8 connection(s)
I20260924 19:00:55.419492 8873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:00:55.421260 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 8873
I20260924 19:00:55.421430 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/wal/instance
I20260924 19:00:55.421907 388 external_mini_cluster.cc:1428] Setting key 1c43858b60231e7da79a4d7665a5726a
Sep 24 19:00:55 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:55.440160 8869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:55.422269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46729 (local address 127.0.97.62:42585)
0924 19:00:55.422483 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:55.422489 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:55.423848 (+ 1359us) server_negotiation.cc:407] Connection header received
0924 19:00:55.425778 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:55.425783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:55.425875 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:55.425994 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:55.427482 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:55.428207 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:55.429903 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:55.430319 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:55.430453 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:55.434502 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:55.434508 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:55.434511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:55.434541 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:55.437052 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:55.437829 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:55.437839 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:55.437841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:55.437955 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:55.438375 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:55.438380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:55.438383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:55.438659 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:55.438807 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:55.439597 (+ 790us) server_negotiation.cc:299] Negotiation successful
0924 19:00:55.439813 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 19:00:55.441149 9003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:55.422879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.2:46729)
0924 19:00:55.423695 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:55.423743 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:55.425451 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:55.426241 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:55.426249 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:55.426750 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:55.427317 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:55.427335 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:55.428358 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:55.428363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:55.429760 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:55.429771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:55.430542 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:55.430548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:55.430891 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:55.431582 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:55.431620 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:55.434273 (+ 2653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:55.437192 (+ 2919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:55.437201 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:55.437217 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:55.437707 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:55.438067 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:55.438073 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:55.438076 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:55.438257 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:55.438784 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:55.438792 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:55.439227 (+ 435us) client_negotiation.cc:769] Sending connection context
0924 19:00:55.439489 (+ 262us) client_negotiation.cc:240] Negotiation successful
0924 19:00:55.440030 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":354,"threads_started":1}
I20260924 19:00:55.442688 9001 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42585
I20260924 19:00:55.442981 9001 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 19:00:55.443687 9001 heartbeater.cc:511] Master 127.0.97.62:42585 requested a full tablet report, sending...
I20260924 19:00:55.445055 8678 ts_manager.cc:194] Registered new tserver with Master: 6dc95019870b46e2a668db848fa2e472 (127.0.97.2:42357)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260924 19:00:55.446137 8678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46729
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 19:00:55.470058 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42585
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:55.785868 9006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:55.786139 9006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:55.786283 9006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:55.793881 9006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:55.794013 9006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:55.794127 9006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:00:55.804201 9006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:55.805406 9006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:55.806782 9006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:55.819355 9012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:55.819363 9014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:55.819463 9011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:55.820477 9006 server_base.cc:1034] running on GCE node
I20260924 19:00:55.821056 9006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:55.822016 9006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:55.823251 9006 hybrid_clock.cc:648] HybridClock initialized: now 1790276455823200 us; error 39 us; skew 500 ppm
I20260924 19:00:55.823553 9006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:55 dist-test-slave-9fn1 krb5kdc[8633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:55.826655 9006 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 19:00:55.828059 9006 webserver.cc:460] Webserver started at http://127.0.97.3:33905/ using document root <none> and password file <none>
I20260924 19:00:55.828663 9006 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:55.828763 9006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:55.829013 9006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:00:55.831823 9006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/data/instance:
uuid: "574fbed571b849f9a4b493c04efeef88"
format_stamp: "Formatted at 2026-09-24 19:00:55 on dist-test-slave-9fn1"
server_key: "8b7465357b6e3a793fe266b1b97b3d6c"
server_key_iv: "e9184f09d8dd6efe5c2358a06095611f"
server_key_version: "encryptionkey@0"
I20260924 19:00:55.832540 9006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/wal/instance:
uuid: "574fbed571b849f9a4b493c04efeef88"
format_stamp: "Formatted at 2026-09-24 19:00:55 on dist-test-slave-9fn1"
server_key: "8b7465357b6e3a793fe266b1b97b3d6c"
server_key_iv: "e9184f09d8dd6efe5c2358a06095611f"
server_key_version: "encryptionkey@0"
I20260924 19:00:55.837595 9006 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260924 19:00:55.840813 9021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:55.842197 9006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:00:55.842374 9006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "574fbed571b849f9a4b493c04efeef88"
format_stamp: "Formatted at 2026-09-24 19:00:55 on dist-test-slave-9fn1"
server_key: "8b7465357b6e3a793fe266b1b97b3d6c"
server_key_iv: "e9184f09d8dd6efe5c2358a06095611f"
server_key_version: "encryptionkey@0"
I20260924 19:00:55.842584 9006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:55.876457 9006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:55.880437 9006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:55.880713 9006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:55.881712 9006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:55.882997 9006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:55.883098 9006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:55.883249 9006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:55.883320 9006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:55.916258 9006 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40031
I20260924 19:00:55.916303 9133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40031 every 8 connection(s)
I20260924 19:00:55.917582 9006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:00:55.918637 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 9006
I20260924 19:00:55.918797 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/wal/instance
I20260924 19:00:55.919369 388 external_mini_cluster.cc:1428] Setting key a15e4f1f5144105315c84c9b93511746
Sep 24 19:00:55 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:00:55.937942 8869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:55.920437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53713 (local address 127.0.97.62:42585)
0924 19:00:55.920638 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:55.920643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:55.922045 (+ 1402us) server_negotiation.cc:407] Connection header received
0924 19:00:55.923939 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:55.923943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:55.924002 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:55.924081 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:55.925618 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:55.926356 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:55.927897 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:55.928281 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:55.928450 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:55.931859 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:55.931865 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:55.931870 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:55.931912 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:55.935087 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:55.935769 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:55.935773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:55.935775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:55.935897 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:55.936387 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:55.936391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:55.936393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:55.936679 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:55.936798 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:55.937479 (+ 681us) server_negotiation.cc:299] Negotiation successful
0924 19:00:55.937705 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 19:00:55.938822 9136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:55.921034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.3:53713)
0924 19:00:55.921860 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:55.921930 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:55.923701 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:55.924244 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:55.924250 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:55.924723 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:55.925418 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:55.925437 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:55.926518 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:55.926523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:55.927737 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:55.927748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:55.928521 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:55.928530 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:55.928814 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:55.929582 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:55.929620 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:55.931655 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:00:55.935199 (+ 3544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:55.935205 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:55.935216 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:55.935627 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:55.936056 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:00:55.936060 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:00:55.936062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:00:55.936242 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:00:55.936799 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:00:55.936805 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:00:55.937109 (+ 304us) client_negotiation.cc:769] Sending connection context
0924 19:00:55.937329 (+ 220us) client_negotiation.cc:240] Negotiation successful
0924 19:00:55.937805 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":335,"threads_started":1}
I20260924 19:00:55.940161 9134 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42585
I20260924 19:00:55.940500 9134 heartbeater.cc:464] Registering TS with master...
I20260924 19:00:55.941179 9134 heartbeater.cc:511] Master 127.0.97.62:42585 requested a full tablet report, sending...
I20260924 19:00:55.942649 8678 ts_manager.cc:194] Registered new tserver with Master: 574fbed571b849f9a4b493c04efeef88 (127.0.97.3:40031)
I20260924 19:00:55.943356 8678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:53713
I20260924 19:00:55.947017 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:00:55.958935 8867 heartbeater.cc:503] Master 127.0.97.62:42585 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260924 19:00:56.080086 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 8739
Sep 24 19:00:56 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 19:00:56.096058 9144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:56.081090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45772 (local address 127.0.97.1:46465)
0924 19:00:56.081751 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:56.081795 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:56.081829 (+ 34us) server_negotiation.cc:407] Connection header received
0924 19:00:56.082021 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:56.082028 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:56.082237 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:56.082415 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:56.083700 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:56.084772 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:56.086122 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:56.086681 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:56.086818 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:56.090758 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:56.090770 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:56.090790 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:56.090827 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:56.093332 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:56.094050 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:56.094062 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:56.094064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:56.094181 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:56.094535 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:56.094540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:56.094542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:56.094853 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:56.095145 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:56.095452 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 19:00:56.095716 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":269,"threads_started":1}
I20260924 19:00:56.097321 8862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:56.178567 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 8739
I20260924 19:00:56.195794 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 8873
Sep 24 19:00:56 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 19:00:56.211146 9146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:56.196694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43600 (local address 127.0.97.2:42357)
0924 19:00:56.197220 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:56.197227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:56.197251 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:00:56.197338 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:56.197350 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:56.197645 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:56.197781 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:56.199011 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:56.199978 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:56.201186 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:56.201825 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:56.201930 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:56.205866 (+ 3936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:56.205875 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:56.205895 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:56.205925 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:56.208470 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:56.209097 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:56.209104 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:56.209106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:56.209213 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:56.209617 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:56.209623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:56.209627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:56.209951 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:56.210260 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:56.210564 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 19:00:56.210812 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":278,"threads_started":1}
I20260924 19:00:56.212348 8996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:56.292989 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 8873
I20260924 19:00:56.308621 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 9006
Sep 24 19:00:56 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 19:00:56.324162 9148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:56.309485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49566 (local address 127.0.97.3:40031)
0924 19:00:56.310183 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:56.310189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:56.310219 (+ 30us) server_negotiation.cc:407] Connection header received
0924 19:00:56.310279 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:56.310286 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:56.310492 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:56.310630 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:56.311822 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:56.312806 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:56.314041 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:56.314587 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:56.314694 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:56.318840 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:56.318852 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:56.318866 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:56.318896 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:56.321356 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:56.322192 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:56.322200 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:56.322202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:56.322315 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:56.322694 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:56.322699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:56.322701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:56.323006 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:56.323340 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:56.323576 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 19:00:56.323835 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":326,"threads_started":1}
I20260924 19:00:56.325413 9129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:56.407517 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 9006
I20260924 19:00:56.423055 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 8646
I20260924 19:00:56.424170 8708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:00:56.523104 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 8646
I20260924 19:00:56.543138 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42585
--webserver_interface=127.0.97.62
--webserver_port=35919
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42585
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:00:56.857695 9151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:56.857988 9151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:56.858146 9151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:56.865693 9151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:00:56.865793 9151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:56.865895 9151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:56.865994 9151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:00:56.866073 9151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:00:56.875635 9151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=35919
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:56.876880 9151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:56.878186 9151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:56.889287 9159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:56.889393 9156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:56.889369 9157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:56.890244 9151 server_base.cc:1034] running on GCE node
I20260924 19:00:56.891006 9151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:56.892216 9151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:56.893442 9151 hybrid_clock.cc:648] HybridClock initialized: now 1790276456893406 us; error 49 us; skew 500 ppm
I20260924 19:00:56.893780 9151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:56 dist-test-slave-9fn1 krb5kdc[8633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:56.896648 9151 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:00:56.898053 9151 webserver.cc:460] Webserver started at http://127.0.97.62:35919/ using document root <none> and password file <none>
I20260924 19:00:56.898697 9151 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:56.898854 9151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:56.903772 9151 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 19:00:56.906335 9166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:56.907528 9151 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 19:00:56.907692 9151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "c69e520e097c4d4aa8cc9c6a254b83ae"
format_stamp: "Formatted at 2026-09-24 19:00:54 on dist-test-slave-9fn1"
server_key: "c4fe0a4de0b503f1f5c27208e2307c0b"
server_key_iv: "0b353e1a5ecfc5f5e0db3a38890b4acc"
server_key_version: "encryptionkey@0"
I20260924 19:00:56.908592 9151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:56.945806 9151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:56.949818 9151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:56.950083 9151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:56.968065 9151 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42585
I20260924 19:00:56.968099 9217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42585 every 8 connection(s)
I20260924 19:00:56.969529 9151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:00:56.971851 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 9151
I20260924 19:00:56.972837 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:46465
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:42585
--webserver_port=35215
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:00:56.978529 9218 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:00:56.982458 9218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae: Bootstrap starting.
I20260924 19:00:57.000979 9218 log.cc:824] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae: Log is configured to *not* fsync() on all Append() calls
I20260924 19:00:57.009356 9218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:00:57.009955 9218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae: Bootstrap complete.
I20260924 19:00:57.019210 9218 raft_consensus.cc:348] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69e520e097c4d4aa8cc9c6a254b83ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42585 } }
I20260924 19:00:57.020023 9218 raft_consensus.cc:729] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c69e520e097c4d4aa8cc9c6a254b83ae, State: Initialized, Role: FOLLOWER
I20260924 19:00:57.020625 9218 consensus_queue.cc:260] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69e520e097c4d4aa8cc9c6a254b83ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42585 } }
I20260924 19:00:57.020849 9218 raft_consensus.cc:388] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:00:57.020960 9218 raft_consensus.cc:482] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:00:57.021121 9218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:00:57.024762 9218 raft_consensus.cc:504] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69e520e097c4d4aa8cc9c6a254b83ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42585 } }
I20260924 19:00:57.025251 9218 leader_election.cc:302] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c69e520e097c4d4aa8cc9c6a254b83ae; no voters:
I20260924 19:00:57.025902 9218 leader_election.cc:288] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 19:00:57.026039 9222 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:00:57.027161 9222 raft_consensus.cc:686] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [term 2 LEADER]: Becoming Leader. State: Replica: c69e520e097c4d4aa8cc9c6a254b83ae, State: Running, Role: LEADER
I20260924 19:00:57.027770 9222 consensus_queue.cc:237] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69e520e097c4d4aa8cc9c6a254b83ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42585 } }
I20260924 19:00:57.028121 9218 sys_catalog.cc:564] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:00:57.030558 9223 sys_catalog.cc:455] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c69e520e097c4d4aa8cc9c6a254b83ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69e520e097c4d4aa8cc9c6a254b83ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42585 } } }
I20260924 19:00:57.030951 9223 sys_catalog.cc:458] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:57.030555 9224 sys_catalog.cc:455] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader c69e520e097c4d4aa8cc9c6a254b83ae. Latest consensus state: current_term: 2 leader_uuid: "c69e520e097c4d4aa8cc9c6a254b83ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69e520e097c4d4aa8cc9c6a254b83ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42585 } } }
I20260924 19:00:57.031284 9224 sys_catalog.cc:458] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae [sys.catalog]: This master's current role is: LEADER
I20260924 19:00:57.033326 9228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:00:57.039283 9228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:00:57.040627 9228 catalog_manager.cc:1259] Loaded cluster ID: d86f4bf8125149c48e269c4a7d1174bc
I20260924 19:00:57.040773 9228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:00:57.045404 9228 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:00:57.046804 9228 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c69e520e097c4d4aa8cc9c6a254b83ae: Loaded TSK: 0
I20260924 19:00:57.048244 9228 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 19:00:57.299551 9220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:57.299850 9220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:57.300011 9220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:57.307107 9220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:57.307256 9220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:57.307423 9220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:00:57.316992 9220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:46465
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=35215
--tserver_master_addrs=127.0.97.62:42585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:57.318296 9220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:57.319590 9220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:57.334276 9245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:57.334439 9247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:57.334527 9244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:57.335649 9220 server_base.cc:1034] running on GCE node
I20260924 19:00:57.336421 9220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:57.337652 9220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:57.338907 9220 hybrid_clock.cc:648] HybridClock initialized: now 1790276457338844 us; error 50 us; skew 500 ppm
I20260924 19:00:57.339298 9220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:57 dist-test-slave-9fn1 krb5kdc[8633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276457, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:57.346700 9220 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260924 19:00:57.348402 9220 webserver.cc:460] Webserver started at http://127.0.97.1:35215/ using document root <none> and password file <none>
I20260924 19:00:57.349287 9220 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:57.349436 9220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:57.356137 9220 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 19:00:57.359146 9254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:57.360631 9220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260924 19:00:57.360781 9220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "2f0857b42806474da2808f93862239d2"
format_stamp: "Formatted at 2026-09-24 19:00:54 on dist-test-slave-9fn1"
server_key: "a6a0cac700197806b386c36a39d541b4"
server_key_iv: "9624e6505572f1c474a47e43486741a3"
server_key_version: "encryptionkey@0"
I20260924 19:00:57.362068 9220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:57.377557 9220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:57.382848 9220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:57.383128 9220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:57.384229 9220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:57.385947 9220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:57.386050 9220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:57.386144 9220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:57.386210 9220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:57.427335 9220 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46465
I20260924 19:00:57.427436 9366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46465 every 8 connection(s)
I20260924 19:00:57.428789 9220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:00:57.435765 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 9220
I20260924 19:00:57.436546 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42357
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:42585
--webserver_port=42443
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:00:57 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:00:57 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:00:57.448741 9370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:57.431692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.1:52531)
0924 19:00:57.432509 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:57.432555 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:57.434435 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:57.435891 (+ 1456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:57.435905 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:57.436662 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:57.437482 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:57.437537 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:57.439146 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:57.439154 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:57.441304 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:57.441322 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:57.442396 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:57.442405 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:57.442736 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:57.443602 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:57.443637 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:57.447808 (+ 4171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":364,"threads_started":1}
W20260924 19:00:57.449445 9367 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42585 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:42585: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
W20260924 19:00:57.450150 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:00:57.431644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52531 (local address 127.0.97.62:42585)
0924 19:00:57.432322 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:57.432333 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:57.432678 (+ 345us) server_negotiation.cc:407] Connection header received
0924 19:00:57.434986 (+ 2308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:57.434999 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:57.435366 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:57.435660 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:57.437729 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:57.438966 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:57.441454 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:57.442164 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:57.442306 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:57.448261 (+ 5955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:52531: BlockingRecv error: recv got EOF from 127.0.97.1:52531 (error 108)
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":274,"threads_started":1}
Sep 24 19:00:57 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:00:57 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:00:57.459903 9370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:57.450122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.1:56437)
0924 19:00:57.450380 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:57.450401 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:57.450507 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:57.451066 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:57.451074 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:57.451520 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:57.451736 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:57.451751 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:57.452836 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:57.452844 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:57.454671 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:57.454689 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:57.455403 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:57.455412 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:57.455644 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:57.456263 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:57.456292 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:57.459615 (+ 3323us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134}
W20260924 19:00:57.460059 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:00:57.450330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56437 (local address 127.0.97.62:42585)
0924 19:00:57.450726 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:57.450731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:57.450749 (+ 18us) server_negotiation.cc:407] Connection header received
0924 19:00:57.450804 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:57.450808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:57.450888 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:57.451022 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:57.451879 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:57.452684 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:57.454833 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:57.455243 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:57.455343 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:57.459813 (+ 4470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56437: BlockingRecv error: recv got EOF from 127.0.97.1:56437 (error 108)
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":1079,"threads_started":1}
Sep 24 19:00:57 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:00:57 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:00:57.468782 9370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:57.460904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.1:48241)
0924 19:00:57.461107 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:57.461156 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:57.461289 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:57.461785 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:57.461792 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:57.462077 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:57.462209 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:57.462216 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:57.463173 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:57.463178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:57.464149 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:57.464160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:57.464848 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:57.464853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:57.465022 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:57.465628 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:57.465644 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:57.468471 (+ 2827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 19:00:57.468802 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:00:57.461310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48241 (local address 127.0.97.62:42585)
0924 19:00:57.461489 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:57.461493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:57.461542 (+ 49us) server_negotiation.cc:407] Connection header received
0924 19:00:57.461586 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:57.461593 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:57.461654 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:57.461754 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:57.462341 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:57.463041 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:57.464292 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:57.464708 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:57.464841 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:57.468621 (+ 3780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48241: BlockingRecv error: recv got EOF from 127.0.97.1:48241 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 19:00:57.469189 9367 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:00:57.771030 9371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:57.771281 9371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:57.771437 9371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:57.778288 9371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:57.778450 9371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:57.778630 9371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:00:57.787854 9371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42357
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=42443
--tserver_master_addrs=127.0.97.62:42585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:57.789068 9371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:57.790365 9371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:57.803526 9378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:57.803462 9377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:57.803500 9380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:57.804207 9371 server_base.cc:1034] running on GCE node
I20260924 19:00:57.804718 9371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:57.805598 9371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:57.806795 9371 hybrid_clock.cc:648] HybridClock initialized: now 1790276457806762 us; error 28 us; skew 500 ppm
I20260924 19:00:57.807102 9371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:57 dist-test-slave-9fn1 krb5kdc[8633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276457, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:57.810052 9371 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260924 19:00:57.811295 9371 webserver.cc:460] Webserver started at http://127.0.97.2:42443/ using document root <none> and password file <none>
I20260924 19:00:57.812017 9371 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:57.812157 9371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:57.817157 9371 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 19:00:57.819687 9387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:57.820912 9371 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 19:00:57.821079 9371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "6dc95019870b46e2a668db848fa2e472"
format_stamp: "Formatted at 2026-09-24 19:00:55 on dist-test-slave-9fn1"
server_key: "3669afa14a0934578db0675c4f8f5840"
server_key_iv: "5475afeba934c7be1737ed14a00e082b"
server_key_version: "encryptionkey@0"
I20260924 19:00:57.822052 9371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:57.842224 9371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:57.846110 9371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:57.846383 9371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:57.847302 9371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:57.848552 9371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:57.848671 9371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:57.848806 9371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:57.848896 9371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:57.880926 9371 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42357
I20260924 19:00:57.881009 9499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42357 every 8 connection(s)
I20260924 19:00:57.882357 9371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:00:57.890683 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 9371
I20260924 19:00:57.891295 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:40031
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:42585
--webserver_port=33905
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:00:57 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:00:57 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 19:00:57.897176 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:00:57.884812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34949 (local address 127.0.97.62:42585)
0924 19:00:57.884977 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:57.884981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:57.886148 (+ 1167us) server_negotiation.cc:407] Connection header received
0924 19:00:57.887874 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:57.887880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:57.887959 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:57.888087 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:57.889456 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:57.890195 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:57.891839 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:57.892255 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:57.892431 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:57.896937 (+ 4506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34949: BlockingRecv error: recv got EOF from 127.0.97.2:34949 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 19:00:57.897883 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:57.885241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.2:34949)
0924 19:00:57.885997 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:57.886037 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:57.887618 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:57.888316 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:57.888323 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:57.888791 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:57.889292 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:57.889307 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:57.890343 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:57.890350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:57.891705 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:57.891717 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:57.892520 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:57.892527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:57.892738 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:57.893401 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:57.893426 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:57.896764 (+ 3338us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":323,"threads_started":1}
W20260924 19:00:57.898507 9500 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42585 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:42585: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 24 19:00:57 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:00:57 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 19:00:57.907460 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:00:57.899235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52341 (local address 127.0.97.62:42585)
0924 19:00:57.899433 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:57.899441 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:57.899461 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:00:57.899514 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:57.899521 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:57.899598 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:57.899699 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:57.900261 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:57.901429 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:57.902567 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:57.903276 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:57.903417 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:57.907300 (+ 3883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52341: BlockingRecv error: recv got EOF from 127.0.97.2:52341 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 19:00:57.907512 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:57.899069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.2:52341)
0924 19:00:57.899228 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:57.899246 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:57.899344 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:57.899740 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:57.899743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:57.900006 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:57.900125 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:57.900131 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:57.901586 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:57.901590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:57.902433 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:57.902442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:57.903411 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:57.903415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:57.903535 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:57.904089 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:57.904106 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:57.907149 (+ 3043us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
Sep 24 19:00:57 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:00:57 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:00:57.915707 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:57.908189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.2:58983)
0924 19:00:57.908361 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:57.908379 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:57.908470 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:57.908781 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:57.908785 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:57.909060 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:57.909186 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:57.909193 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:57.910283 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:57.910290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:57.911237 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:57.911247 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:57.911920 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:57.911925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:57.912083 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:57.912666 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:57.912693 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:57.915438 (+ 2745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 19:00:57.915741 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:00:57.908279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58983 (local address 127.0.97.62:42585)
0924 19:00:57.908440 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:57.908445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:57.908460 (+ 15us) server_negotiation.cc:407] Connection header received
0924 19:00:57.908596 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:57.908601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:57.908661 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:57.908761 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:57.909312 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:57.910159 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:57.911360 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:57.911785 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:57.911901 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:57.915599 (+ 3698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58983: BlockingRecv error: recv got EOF from 127.0.97.2:58983 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 19:00:57.916081 9500 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:00:58.212383 9503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:00:58.212652 9503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:00:58.212874 9503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:00:58.219930 9503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:00:58.220034 9503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:00:58.220237 9503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:00:58.230118 9503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:40031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=33905
--tserver_master_addrs=127.0.97.62:42585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:00:58.231509 9503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:00:58.232792 9503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:00:58.245705 9508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:58.245764 9509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:00:58.246320 9511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:00:58.246390 9503 server_base.cc:1034] running on GCE node
I20260924 19:00:58.246953 9503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:00:58.247747 9503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:00:58.248966 9503 hybrid_clock.cc:648] HybridClock initialized: now 1790276458248915 us; error 67 us; skew 500 ppm
I20260924 19:00:58.249260 9503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:00:58 dist-test-slave-9fn1 krb5kdc[8633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276458, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:00:58.252416 9503 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260924 19:00:58.253819 9503 webserver.cc:460] Webserver started at http://127.0.97.3:33905/ using document root <none> and password file <none>
I20260924 19:00:58.254424 9503 fs_manager.cc:359] Metadata directory not provided
I20260924 19:00:58.254529 9503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:00:58.259438 9503 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 19:00:58.261987 9518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:58.263262 9503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260924 19:00:58.263429 9503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "574fbed571b849f9a4b493c04efeef88"
format_stamp: "Formatted at 2026-09-24 19:00:55 on dist-test-slave-9fn1"
server_key: "8b7465357b6e3a793fe266b1b97b3d6c"
server_key_iv: "e9184f09d8dd6efe5c2358a06095611f"
server_key_version: "encryptionkey@0"
I20260924 19:00:58.264326 9503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:00:58.291147 9503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:00:58.295377 9503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:00:58.295634 9503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:00:58.296561 9503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:00:58.297890 9503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:00:58.298012 9503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:58.298141 9503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:00:58.298229 9503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:00:58.330538 9503 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40031
I20260924 19:00:58.330621 9630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40031 every 8 connection(s)
I20260924 19:00:58.331915 9503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:00:58.333191 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 9503
Sep 24 19:00:58 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:00:58 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 19:00:58.347934 9634 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:00:58.334601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54709 (local address 127.0.97.62:42585)
0924 19:00:58.335101 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:58.335107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:58.336092 (+ 985us) server_negotiation.cc:407] Connection header received
0924 19:00:58.338150 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:58.338155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:58.338219 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:58.338310 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:58.339725 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:58.340444 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:58.342139 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:58.342563 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:58.342691 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:58.347630 (+ 4939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:54709: BlockingRecv error: recv got EOF from 127.0.97.3:54709 (error 108)
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":382,"thread_start_us":266,"threads_started":1}
I20260924 19:00:58.348503 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:58.335135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.3:54709)
0924 19:00:58.335916 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:58.335965 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:58.337918 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:58.338563 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:58.338571 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:58.339030 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:58.339557 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:58.339572 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:58.340592 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:58.340596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:58.342003 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:58.342015 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:58.342780 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:58.342790 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:58.343011 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:58.343614 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:58.343634 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:58.347401 (+ 3767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":333,"threads_started":1}
W20260924 19:00:58.349195 9631 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42585 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:42585: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260924 19:00:58.349748 9369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:58.334216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39252 (local address 127.0.97.62:42585)
0924 19:00:58.334468 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:58.334478 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:58.336639 (+ 2161us) server_negotiation.cc:407] Connection header received
0924 19:00:58.336756 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:58.336763 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:58.336864 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:58.336972 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:58.338427 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:58.339201 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:58.340439 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:58.340846 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:58.340976 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:58.343482 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:00:58.343490 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:00:58.343504 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:00:58.343565 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:00:58.346638 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:58.347288 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:58.347301 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:58.347307 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:58.347473 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:00:58.347843 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:00:58.347854 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:00:58.347857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:00:58.348379 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:00:58.348665 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0924 19:00:58.349073 (+ 408us) server_negotiation.cc:299] Negotiation successful
0924 19:00:58.349392 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":78}
Sep 24 19:00:58 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:00:58 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:00:58.357760 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:58.349793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.3:33851)
0924 19:00:58.350002 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:58.350020 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:58.350104 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:58.351082 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:58.351087 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:58.351435 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:58.351604 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:58.351614 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:58.352808 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:58.352813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:58.353957 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:58.353968 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:58.354568 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:58.354571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:58.354690 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:58.355153 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:58.355172 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:58.357571 (+ 2399us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125}
W20260924 19:00:58.357933 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:00:58.349875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33851 (local address 127.0.97.62:42585)
0924 19:00:58.350838 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:58.350842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:58.350855 (+ 13us) server_negotiation.cc:407] Connection header received
0924 19:00:58.350906 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:58.350909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:58.350953 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:58.351023 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:58.351752 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:58.352681 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:58.354072 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:58.354456 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:58.354547 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:58.357726 (+ 3179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:33851: BlockingRecv error: recv got EOF from 127.0.97.3:33851 (error 108)
Metrics: {"mutex_wait_us":788,"server-negotiator.queue_time_us":848}
Sep 24 19:00:58 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:00:58 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:00:58.365234 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:58.358348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.3:49685)
0924 19:00:58.358529 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:58.358547 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:58.358634 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:58.358859 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:58.358862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:58.359105 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:58.359230 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:58.359237 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:58.360101 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:58.360104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:58.360943 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:58.360952 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:58.361554 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:58.361559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:58.361669 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:58.362117 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:58.362132 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:58.364962 (+ 2830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 19:00:58.365392 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:00:58.358426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49685 (local address 127.0.97.62:42585)
0924 19:00:58.358618 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:58.358622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:58.358635 (+ 13us) server_negotiation.cc:407] Connection header received
0924 19:00:58.358703 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:58.358706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:58.358749 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:58.358817 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:58.359335 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:58.359994 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:58.361049 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:58.361400 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:58.361492 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:58.365174 (+ 3682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49685: BlockingRecv error: recv got EOF from 127.0.97.3:49685 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
W20260924 19:00:58.365540 9631 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 19:00:58 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:00:58 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:00:58.478005 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:58.469791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.1:37339)
0924 19:00:58.470297 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:58.470312 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:58.470415 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:58.470668 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:58.470672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:58.471091 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:58.471269 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:58.471277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:58.472176 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:58.472179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:58.473719 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:58.473730 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:58.474378 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:58.474382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:58.474498 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:58.475023 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:58.475038 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:58.477780 (+ 2742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":242,"threads_started":1}
W20260924 19:00:58.478142 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:00:58.469833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37339 (local address 127.0.97.62:42585)
0924 19:00:58.470025 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:58.470034 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:58.470381 (+ 347us) server_negotiation.cc:407] Connection header received
0924 19:00:58.470491 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:58.470494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:58.470549 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:58.470622 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:58.471379 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:58.472035 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:58.473842 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:58.474229 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:58.474341 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:58.477955 (+ 3614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37339: BlockingRecv error: recv got EOF from 127.0.97.1:37339 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 19:00:58 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:00:58 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:00:58.924106 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:58.916597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.2:38115)
0924 19:00:58.917116 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:58.917132 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:58.917238 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:58.917555 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:58.917559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:58.917988 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:58.918176 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:58.918184 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:58.919071 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:58.919075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:58.919962 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:58.919972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:58.920602 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:58.920605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:58.920716 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:58.921258 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:58.921274 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:58.923895 (+ 2621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":302,"threads_started":1}
W20260924 19:00:58.924432 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:00:58.916679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38115 (local address 127.0.97.62:42585)
0924 19:00:58.916863 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:58.916867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:58.917201 (+ 334us) server_negotiation.cc:407] Connection header received
0924 19:00:58.917330 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:58.917333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:58.917389 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:58.917467 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:58.918299 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:58.918960 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:58.920076 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:58.920463 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:58.920550 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:58.924155 (+ 3605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38115: BlockingRecv error: recv got EOF from 127.0.97.2:38115 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 19:00:59 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:00:59 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:00:59.374653 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:59.366182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.3:54309)
0924 19:00:59.366708 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:59.366727 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:59.366839 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:59.367115 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:59.367118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:59.367624 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:59.367868 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:59.367877 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:59.368964 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:59.368968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:59.370024 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:59.370035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:59.370673 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:59.370677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:59.370804 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:59.371397 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:59.371413 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:59.374445 (+ 3032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":276,"threads_started":1}
W20260924 19:00:59.374840 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:00:59.366325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54309 (local address 127.0.97.62:42585)
0924 19:00:59.366497 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:59.366501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:59.366799 (+ 298us) server_negotiation.cc:407] Connection header received
0924 19:00:59.366931 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:59.366935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:59.366990 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:59.367069 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:59.367985 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:59.368826 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:59.370143 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:59.370531 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:59.370622 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:59.374603 (+ 3981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:54309: BlockingRecv error: recv got EOF from 127.0.97.3:54309 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 19:00:59 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:00:59 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:00:59.487133 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:59.479085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.1:53765)
0924 19:00:59.479610 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:59.479625 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:59.479794 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:59.480130 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:59.480134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:59.480656 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:59.480805 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:59.480814 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:59.481896 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:59.481902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:59.482810 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:59.482821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:59.483521 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:59.483526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:59.483687 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:59.484244 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:59.484260 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:59.486890 (+ 2630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":296,"threads_started":1}
W20260924 19:00:59.487319 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:00:59.479135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53765 (local address 127.0.97.62:42585)
0924 19:00:59.479332 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:59.479338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:59.479717 (+ 379us) server_negotiation.cc:407] Connection header received
0924 19:00:59.479917 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:59.479923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:59.479993 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:59.480099 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:59.480947 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:59.481756 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:59.482951 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:59.483370 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:59.483506 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:59.487068 (+ 3562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53765: BlockingRecv error: recv got EOF from 127.0.97.1:53765 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 19:00:59 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:00:59 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:00:59.933832 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:00:59.925358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.2:53129)
0924 19:00:59.925885 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0924 19:00:59.925899 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:00:59.926038 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:00:59.926363 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:00:59.926366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:00:59.926847 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:00:59.927021 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:59.927031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:59.928286 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:59.928291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:59.929256 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:00:59.929266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:00:59.930060 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:00:59.930066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:00:59.930237 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:59.930792 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:00:59.930816 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:00:59.933603 (+ 2787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":337,"threads_started":1}
W20260924 19:00:59.934017 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:00:59.925438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53129 (local address 127.0.97.62:42585)
0924 19:00:59.925708 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 19:00:59.925716 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:00:59.925978 (+ 262us) server_negotiation.cc:407] Connection header received
0924 19:00:59.926135 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:00:59.926139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:00:59.926223 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:00:59.926344 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:00:59.927185 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:59.928102 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:59.929415 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:00:59.929906 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:00:59.930041 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:00:59.933818 (+ 3777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:53129: BlockingRecv error: recv got EOF from 127.0.97.2:53129 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 24 19:01:00 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:00 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:00.389684 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:00.375631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.3:47091)
0924 19:01:00.376157 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:00.376186 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:00.376338 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:00.376668 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:00.376672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:00.377105 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:00.377238 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:00.377247 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:00.378316 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:00.378323 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:00.379244 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:00.379254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:00.379960 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:00.379966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:00.380137 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:00.386152 (+ 6015us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:00.386170 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:00.389339 (+ 3169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":330,"threads_started":1}
W20260924 19:01:00.389786 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:00.375753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47091 (local address 127.0.97.62:42585)
0924 19:01:00.375965 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:00.375972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:00.376274 (+ 302us) server_negotiation.cc:407] Connection header received
0924 19:01:00.376459 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:00.376464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:00.376534 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:00.376642 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:00.377385 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:00.378166 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:00.379388 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:00.379802 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:00.379931 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:00.389544 (+ 9613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47091: BlockingRecv error: recv got EOF from 127.0.97.3:47091 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 19:01:00 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:00 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:00.496248 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:00.488007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.1:37585)
0924 19:01:00.488456 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:00.488472 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:00.488585 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:00.488874 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:00.488877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:00.489224 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:00.489392 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:00.489401 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:00.490441 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:00.490445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:00.491352 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:00.491364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:00.492018 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:00.492022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:00.492143 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:00.492794 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:00.492811 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:00.495882 (+ 3071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":266,"threads_started":1}
W20260924 19:01:00.496279 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:00.488100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37585 (local address 127.0.97.62:42585)
0924 19:01:00.488333 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:00.488340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:00.488555 (+ 215us) server_negotiation.cc:407] Connection header received
0924 19:01:00.488686 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:00.488690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:00.488750 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:00.488829 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:00.489536 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:00.490316 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:00.491478 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:00.491892 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:00.491985 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:00.496061 (+ 4076us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37585: BlockingRecv error: recv got EOF from 127.0.97.1:37585 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 24 19:01:00 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:00 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:00.942220 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:00.934614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.2:36385)
0924 19:01:00.935173 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:00.935199 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:00.935306 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:00.935593 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:00.935597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:00.935909 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:00.936061 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:00.936069 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:00.937007 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:00.937010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:00.937913 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:00.937924 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:00.938525 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:00.938529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:00.938652 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:00.939197 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:00.939213 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:00.941848 (+ 2635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":346,"threads_started":1}
W20260924 19:01:00.942271 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:00.934725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36385 (local address 127.0.97.62:42585)
0924 19:01:00.934939 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:00.934944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:00.935269 (+ 325us) server_negotiation.cc:407] Connection header received
0924 19:01:00.935403 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:00.935407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:00.935465 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:00.935552 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:00.936184 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:00.936894 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:00.938033 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:00.938412 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:00.938500 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:00.942032 (+ 3532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36385: BlockingRecv error: recv got EOF from 127.0.97.2:36385 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 19:01:01 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:01 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:01.398270 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:01.390600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.3:49829)
0924 19:01:01.391016 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:01.391035 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:01.391139 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:01.391385 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:01.391388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:01.391691 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:01.391840 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:01.391847 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:01.392739 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:01.392742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:01.393704 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:01.393721 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:01.394388 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:01.394392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:01.394506 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:01.395029 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:01.395045 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:01.397999 (+ 2954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":239,"threads_started":1}
W20260924 19:01:01.398434 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:01.390691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49829 (local address 127.0.97.62:42585)
0924 19:01:01.390924 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:01.390930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:01.391109 (+ 179us) server_negotiation.cc:407] Connection header received
0924 19:01:01.391229 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:01.391232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:01.391300 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:01.391375 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:01.391958 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:01.392627 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:01.393847 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:01.394267 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:01.394361 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:01.398222 (+ 3861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:49829: BlockingRecv error: recv got EOF from 127.0.97.3:49829 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 24 19:01:01 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:01 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:01.504671 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:01.497097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.1:43127)
0924 19:01:01.497619 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:01.497634 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:01.497742 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:01.498029 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:01.498032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:01.498378 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:01.498553 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:01.498561 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:01.499497 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:01.499500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:01.500383 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:01.500393 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:01.500988 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:01.500992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:01.501111 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:01.501765 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:01.501781 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:01.504446 (+ 2665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":287,"threads_started":1}
W20260924 19:01:01.504848 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:01.497289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43127 (local address 127.0.97.62:42585)
0924 19:01:01.497459 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:01.497463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:01.497701 (+ 238us) server_negotiation.cc:407] Connection header received
0924 19:01:01.497854 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:01.497858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:01.497914 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:01.497984 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:01.498676 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:01.499390 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:01.500491 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:01.500858 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:01.500961 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:01.504595 (+ 3634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43127: BlockingRecv error: recv got EOF from 127.0.97.1:43127 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 19:01:01 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:01 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:01.951555 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:01.943199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.2:40615)
0924 19:01:01.943673 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:01.943688 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:01.943846 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:01.944175 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:01.944179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:01.944560 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:01.944697 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:01.944706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:01.945761 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:01.945768 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:01.946720 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:01.946730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:01.947605 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:01.947609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:01.947911 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:01.948606 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:01.948628 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:01.951327 (+ 2699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":252,"threads_started":1}
W20260924 19:01:01.951748 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:01.943273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40615 (local address 127.0.97.62:42585)
0924 19:01:01.943555 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:01.943561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:01.943774 (+ 213us) server_negotiation.cc:407] Connection header received
0924 19:01:01.943977 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:01.943982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:01.944048 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:01.944150 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:01.944846 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:01.945634 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:01.946869 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:01.947438 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:01.947588 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:01.951530 (+ 3942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40615: BlockingRecv error: recv got EOF from 127.0.97.2:40615 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 24 19:01:02 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:02 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:02.407047 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:02.399110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.3:35323)
0924 19:01:02.399558 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:02.399577 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:02.399723 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:02.400069 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:02.400074 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:02.400455 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:02.400598 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:02.400606 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:02.401613 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:02.401622 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:02.402591 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:02.402602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:02.403289 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:02.403294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:02.403471 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:02.404101 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:02.404117 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:02.406728 (+ 2611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":240,"threads_started":1}
W20260924 19:01:02.407162 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:02.399181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35323 (local address 127.0.97.62:42585)
0924 19:01:02.399406 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:02.399414 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:02.399658 (+ 244us) server_negotiation.cc:407] Connection header received
0924 19:01:02.399852 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:02.399857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:02.399939 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:02.400039 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:02.400746 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:02.401453 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:02.402737 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:02.403148 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:02.403279 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:02.406919 (+ 3640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35323: BlockingRecv error: recv got EOF from 127.0.97.3:35323 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 24 19:01:02 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:02 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:02.514434 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:02.505601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.1:51737)
0924 19:01:02.506088 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:02.506103 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:02.506253 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:02.506597 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:02.506601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:02.507011 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:02.507172 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:02.507181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:02.508176 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:02.508182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:02.509099 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:02.509109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:02.509869 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:02.509874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:02.510050 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:02.510677 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:02.510694 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:02.514093 (+ 3399us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":228,"threads_started":1}
W20260924 19:01:02.514526 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:02.505637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51737 (local address 127.0.97.62:42585)
0924 19:01:02.505840 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:02.505848 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:02.506187 (+ 339us) server_negotiation.cc:407] Connection header received
0924 19:01:02.506388 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:02.506394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:02.506461 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:02.506563 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:02.507327 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:02.508035 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:02.509238 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:02.509723 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:02.509858 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:02.514272 (+ 4414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51737: BlockingRecv error: recv got EOF from 127.0.97.1:51737 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 19:01:02 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:02 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:02.960141 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:02.952512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.2:52299)
0924 19:01:02.953031 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:02.953046 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:02.953149 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:02.953425 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:02.953428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:02.953881 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:02.954032 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:02.954040 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:02.954989 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:02.954992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:02.955963 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:02.955973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:02.956579 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:02.956583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:02.956715 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:02.957261 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:02.957277 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:02.959865 (+ 2588us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":259,"threads_started":1}
W20260924 19:01:02.960330 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:02.952642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52299 (local address 127.0.97.62:42585)
0924 19:01:02.952854 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:02.952858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:02.953116 (+ 258us) server_negotiation.cc:407] Connection header received
0924 19:01:02.953243 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:02.953247 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:02.953301 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:02.953384 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:02.954149 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:02.954879 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:02.956082 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:02.956469 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:02.956556 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:02.960070 (+ 3514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52299: BlockingRecv error: recv got EOF from 127.0.97.2:52299 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 19:01:03 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:03 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:03.415380 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:03.407969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.3:55377)
0924 19:01:03.408486 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:03.408505 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:03.408607 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:03.408835 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:03.408837 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:03.409169 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:03.409321 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:03.409330 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:03.410273 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:03.410277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:03.411222 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:03.411233 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:03.411910 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:03.411914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:03.412016 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:03.412541 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:03.412556 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:03.415146 (+ 2590us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":262,"threads_started":1}
W20260924 19:01:03.415542 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:03.408015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55377 (local address 127.0.97.62:42585)
0924 19:01:03.408163 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:03.408167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:03.408568 (+ 401us) server_negotiation.cc:407] Connection header received
0924 19:01:03.408692 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:03.408695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:03.408748 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:03.408820 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:03.409428 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:03.410160 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:03.411345 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:03.411770 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:03.411917 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:03.415306 (+ 3389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:55377: BlockingRecv error: recv got EOF from 127.0.97.3:55377 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 19:01:03 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:03 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:03.523078 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:03.515516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.1:58237)
0924 19:01:03.516025 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:03.516041 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:03.516151 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:03.516429 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:03.516432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:03.516793 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:03.516964 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:03.516972 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:03.517990 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:03.517994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:03.518905 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:03.518915 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:03.519587 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:03.519591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:03.519709 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:03.520273 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:03.520290 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:03.522876 (+ 2586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":295,"threads_started":1}
W20260924 19:01:03.523289 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:03.515613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58237 (local address 127.0.97.62:42585)
0924 19:01:03.515778 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:03.515782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:03.516109 (+ 327us) server_negotiation.cc:407] Connection header received
0924 19:01:03.516249 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:03.516252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:03.516308 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:03.516387 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:03.517089 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:03.517871 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:03.519027 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:03.519436 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:03.519536 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:03.523063 (+ 3527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58237: BlockingRecv error: recv got EOF from 127.0.97.1:58237 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 19:01:03 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:03 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:03.969208 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:03.961332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.2:33073)
0924 19:01:03.961879 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:03.961894 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:03.961998 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:03.962260 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:03.962263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:03.962603 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:03.962755 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:03.962762 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:03.963637 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:03.963640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:03.964476 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:03.964485 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:03.965206 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:03.965215 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:03.965328 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:03.965918 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:03.965934 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:03.968905 (+ 2971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":305,"threads_started":1}
W20260924 19:01:03.969344 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:03.961411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33073 (local address 127.0.97.62:42585)
0924 19:01:03.961624 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:03.961629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:03.961964 (+ 335us) server_negotiation.cc:407] Connection header received
0924 19:01:03.962092 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:03.962095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:03.962150 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:03.962217 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:03.962872 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:03.963528 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:03.964606 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:03.964972 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:03.965153 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:03.969090 (+ 3937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:33073: BlockingRecv error: recv got EOF from 127.0.97.2:33073 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 19:01:04 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:04 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:04.424266 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:04.416442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.3:36615)
0924 19:01:04.417016 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:04.417036 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:04.417145 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:04.417486 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:04.417491 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:04.417876 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:04.418035 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:04.418043 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:04.419129 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:04.419133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:04.420006 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:04.420016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:04.420743 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:04.420747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:04.420863 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:04.421409 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:04.421425 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:04.424041 (+ 2616us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":305,"threads_started":1}
W20260924 19:01:04.424422 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:04.416547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36615 (local address 127.0.97.62:42585)
0924 19:01:04.416783 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:04.416792 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:04.417112 (+ 320us) server_negotiation.cc:407] Connection header received
0924 19:01:04.417261 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:04.417266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:04.417344 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:04.417442 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:04.418202 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:04.418984 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:04.420135 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:04.420617 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:04.420710 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:04.424219 (+ 3509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:36615: BlockingRecv error: recv got EOF from 127.0.97.3:36615 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 24 19:01:04 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:04 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:04.532225 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:04.523957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.1:58375)
0924 19:01:04.524493 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:04.524509 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:04.524664 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:04.525015 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:04.525020 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:04.525589 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:04.525758 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:04.525767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:04.526867 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:04.526878 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:04.527774 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:04.527785 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:04.528567 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:04.528572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:04.528738 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:04.529272 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:04.529291 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:04.532010 (+ 2719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":224,"threads_started":1}
W20260924 19:01:04.532387 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:04.523998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58375 (local address 127.0.97.62:42585)
0924 19:01:04.524202 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:04.524211 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:04.524593 (+ 382us) server_negotiation.cc:407] Connection header received
0924 19:01:04.524808 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:04.524814 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:04.524885 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:04.524985 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:04.525896 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:04.526718 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:04.527917 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:04.528425 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:04.528555 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:04.532194 (+ 3639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58375: BlockingRecv error: recv got EOF from 127.0.97.1:58375 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 24 19:01:04 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:04 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:04.978734 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:04.970264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.2:38913)
0924 19:01:04.970850 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:04.970866 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:04.970975 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:04.971287 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:04.971290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:04.971680 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:04.971844 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:04.971852 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:04.973216 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:04.973220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:04.974155 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:04.974165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:04.975131 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:04.975136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:04.975240 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:04.975861 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:04.975877 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:04.978510 (+ 2633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":254,"threads_started":1}
W20260924 19:01:04.978971 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:04.970349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38913 (local address 127.0.97.62:42585)
0924 19:01:04.970599 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:04.970605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:04.970944 (+ 339us) server_negotiation.cc:407] Connection header received
0924 19:01:04.971059 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:04.971063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:04.971154 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:04.971254 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:04.971981 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:04.973085 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:04.974279 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:04.974989 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:04.975105 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:04.978709 (+ 3604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38913: BlockingRecv error: recv got EOF from 127.0.97.2:38913 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 24 19:01:05 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:05 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:05.439479 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:05.425363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.3:43959)
0924 19:01:05.426178 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:05.426198 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:05.426311 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:05.426613 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:05.426616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:05.427075 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:05.427288 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:05.427298 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:05.428671 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:05.428679 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:05.429864 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:05.429876 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:05.430631 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:05.430654 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:05.430815 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:05.436304 (+ 5489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:05.436321 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:05.439156 (+ 2835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":249,"threads_started":1}
W20260924 19:01:05.439628 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:05.425448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43959 (local address 127.0.97.62:42585)
0924 19:01:05.425685 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:05.425693 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:05.426278 (+ 585us) server_negotiation.cc:407] Connection header received
0924 19:01:05.426436 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:05.426440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:05.426499 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:05.426575 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:05.427429 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:05.428555 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:05.430017 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:05.430450 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:05.430617 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:05.439368 (+ 8751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43959: BlockingRecv error: recv got EOF from 127.0.97.3:43959 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 24 19:01:05 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:05 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:05.541392 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:05.533224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.1:55695)
0924 19:01:05.533885 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:05.533900 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:05.534043 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:05.534410 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:05.534414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:05.534938 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:05.535085 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:05.535094 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:05.536134 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:05.536141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:05.537026 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:05.537037 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:05.537836 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:05.537842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:05.538006 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:05.538553 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:05.538569 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:05.541165 (+ 2596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":428,"threads_started":1}
W20260924 19:01:05.541744 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:05.533415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55695 (local address 127.0.97.62:42585)
0924 19:01:05.533628 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:05.533649 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:05.533975 (+ 326us) server_negotiation.cc:407] Connection header received
0924 19:01:05.534175 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:05.534198 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:05.534273 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:05.534390 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:05.535225 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:05.535992 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:05.537163 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:05.537664 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:05.537802 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:05.541347 (+ 3545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55695: BlockingRecv error: recv got EOF from 127.0.97.1:55695 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 19:01:05 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:05 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:05.987974 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:05.979700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.2:37111)
0924 19:01:05.980174 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:05.980189 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:05.980355 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:05.980695 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:05.980700 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:05.981107 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:05.981250 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:05.981258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:05.982486 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:05.982492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:05.983385 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:05.983395 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:05.984144 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:05.984151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:05.984331 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:05.984934 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:05.984950 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:05.987657 (+ 2707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":250,"threads_started":1}
W20260924 19:01:05.988107 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:05.979790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37111 (local address 127.0.97.62:42585)
0924 19:01:05.980021 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:05.980026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:05.980274 (+ 248us) server_negotiation.cc:407] Connection header received
0924 19:01:05.980480 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:05.980485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:05.980566 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:05.980662 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:05.981391 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:05.982345 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:05.983530 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:05.983962 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:05.984087 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:05.987873 (+ 3786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:37111: BlockingRecv error: recv got EOF from 127.0.97.2:37111 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 24 19:01:06 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:06 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:06.448594 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:06.440491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.3:58533)
0924 19:01:06.441000 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:06.441019 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:06.441165 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:06.441615 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:06.441623 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:06.442143 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:06.442395 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:06.442408 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:06.443404 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:06.443408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:06.444327 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:06.444337 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:06.445021 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:06.445025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:06.445138 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:06.445753 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:06.445768 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:06.448368 (+ 2600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":277,"threads_started":1}
W20260924 19:01:06.448814 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:06.440600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58533 (local address 127.0.97.62:42585)
0924 19:01:06.440850 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:06.440856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:06.441264 (+ 408us) server_negotiation.cc:407] Connection header received
0924 19:01:06.441326 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:06.441329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:06.441408 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:06.441536 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:06.442546 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:06.443265 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:06.444464 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:06.444855 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:06.444971 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:06.448571 (+ 3600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58533: BlockingRecv error: recv got EOF from 127.0.97.3:58533 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 24 19:01:06 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:06 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:06.550895 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:06.542344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.1:44995)
0924 19:01:06.543156 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:06.543171 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:06.543310 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:06.543585 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:06.543588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:06.543958 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:06.544118 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:06.544126 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:06.545093 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:06.545098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:06.546073 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:06.546084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:06.546756 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:06.546760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:06.546880 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:06.547499 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:06.547517 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:06.550643 (+ 3126us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":537,"threads_started":1}
W20260924 19:01:06.551023 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:06.542424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44995 (local address 127.0.97.62:42585)
0924 19:01:06.542680 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:06.542699 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:06.543282 (+ 583us) server_negotiation.cc:407] Connection header received
0924 19:01:06.543404 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:06.543408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:06.543466 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:06.543546 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:06.544240 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:06.544970 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:06.546193 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:06.546604 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:06.546697 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:06.550815 (+ 4118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:44995: BlockingRecv error: recv got EOF from 127.0.97.1:44995 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 24 19:01:06 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:06 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:06.996627 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:06.988926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.2:32923)
0924 19:01:06.989386 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:06.989402 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:06.989571 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:06.989851 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:06.989854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:06.990228 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:06.990392 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:06.990400 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:06.991332 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:06.991336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:06.992261 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:06.992271 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:06.992939 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:06.992943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:06.993056 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:06.993660 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:06.993676 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:06.996394 (+ 2718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":258,"threads_started":1}
W20260924 19:01:06.996886 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:06.989012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32923 (local address 127.0.97.62:42585)
0924 19:01:06.989257 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:06.989264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:06.989483 (+ 219us) server_negotiation.cc:407] Connection header received
0924 19:01:06.989685 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:06.989688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:06.989756 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:06.989834 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:06.990508 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:06.991210 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:06.992387 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:06.992809 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:06.992900 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:06.996607 (+ 3707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:32923: BlockingRecv error: recv got EOF from 127.0.97.2:32923 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 24 19:01:07 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:07 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:07.457295 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:07.449506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.3:35863)
0924 19:01:07.450092 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:07.450118 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:07.450222 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:07.450522 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:07.450525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:07.450885 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:07.451048 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:07.451056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:07.452001 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:07.452005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:07.452893 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:07.452903 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:07.453572 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:07.453576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:07.453690 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:07.454236 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:07.454252 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:07.456951 (+ 2699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":305,"threads_started":1}
W20260924 19:01:07.457345 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:07.449576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35863 (local address 127.0.97.62:42585)
0924 19:01:07.449732 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:07.449737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:07.450199 (+ 462us) server_negotiation.cc:407] Connection header received
0924 19:01:07.450342 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:07.450346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:07.450402 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:07.450477 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:07.451179 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:07.451897 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:07.453012 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:07.453429 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:07.453588 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:07.457132 (+ 3544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:35863: BlockingRecv error: recv got EOF from 127.0.97.3:35863 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 19:01:07 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:07 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:07.559793 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:07.551883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.1:56511)
0924 19:01:07.552400 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:07.552416 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:07.552554 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:07.552880 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:07.552884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:07.553266 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:07.553414 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:07.553422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:07.554497 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:07.554505 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:07.555431 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:07.555445 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:07.556121 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:07.556127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:07.556306 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:07.556917 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:07.556933 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:07.559578 (+ 2645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":321,"threads_started":1}
W20260924 19:01:07.560070 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:07.551993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56511 (local address 127.0.97.62:42585)
0924 19:01:07.552173 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:07.552178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:07.552501 (+ 323us) server_negotiation.cc:407] Connection header received
0924 19:01:07.552694 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:07.552698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:07.552753 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:07.552851 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:07.553570 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:07.554369 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:07.555573 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:07.555967 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:07.556092 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:07.559750 (+ 3658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56511: BlockingRecv error: recv got EOF from 127.0.97.1:56511 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 19:01:07 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:07 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:08.005767 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:07.997544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.2:35381)
0924 19:01:07.998069 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:07.998084 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:07.998236 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:07.998574 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:07.998579 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:07.998967 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:07.999096 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:07.999104 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:08.000066 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:08.000072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:08.001103 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:08.001135 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:08.002095 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:08.002101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:08.002272 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:08.002940 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:08.002958 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:08.005549 (+ 2591us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":292,"threads_started":1}
W20260924 19:01:08.005937 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:07.997705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35381 (local address 127.0.97.62:42585)
0924 19:01:07.997932 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:07.997939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:07.998168 (+ 229us) server_negotiation.cc:407] Connection header received
0924 19:01:07.998383 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:07.998387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:07.998443 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:07.998551 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:07.999239 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:07.999927 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:08.001265 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:08.001897 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:08.002031 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:08.005740 (+ 3709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:35381: BlockingRecv error: recv got EOF from 127.0.97.2:35381 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 19:01:08 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:08 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:08.466473 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:08.458461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.3:59523)
0924 19:01:08.458939 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:08.458966 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:08.459100 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:08.459486 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:08.459491 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:08.459952 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:08.460094 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:08.460103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:08.461045 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:08.461049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:08.462046 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:08.462057 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:08.462773 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:08.462777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:08.462892 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:08.463438 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:08.463453 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:08.466184 (+ 2731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":322,"threads_started":1}
W20260924 19:01:08.466539 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:08.458604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59523 (local address 127.0.97.62:42585)
0924 19:01:08.458787 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:08.458793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:08.459074 (+ 281us) server_negotiation.cc:407] Connection header received
0924 19:01:08.459251 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:08.459258 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:08.459335 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:08.459418 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:08.460235 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:08.460920 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:08.462162 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:08.462593 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:08.462724 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:08.466367 (+ 3643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59523: BlockingRecv error: recv got EOF from 127.0.97.3:59523 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 19:01:08 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:08 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:08.570214 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:08.560844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.1:46949)
0924 19:01:08.561349 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:08.561364 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:08.561542 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:08.561878 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:08.561883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:08.562436 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:08.562730 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:08.562742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:08.564216 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:08.564220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:08.565428 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:08.565440 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:08.566302 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:08.566308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:08.566473 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:08.567065 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:08.567081 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:08.569964 (+ 2883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":284,"threads_started":1}
W20260924 19:01:08.570382 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:08.560951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46949 (local address 127.0.97.62:42585)
0924 19:01:08.561174 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:08.561181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:08.561454 (+ 273us) server_negotiation.cc:407] Connection header received
0924 19:01:08.561676 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:08.561681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:08.561752 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:08.561854 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:08.562890 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:08.563930 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:08.565616 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:08.566164 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:08.566280 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:08.570160 (+ 3880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:46949: BlockingRecv error: recv got EOF from 127.0.97.1:46949 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 24 19:01:09 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:09 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:09.015286 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:09.006834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.2:47985)
0924 19:01:09.007311 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:09.007326 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:09.007481 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:09.007815 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:09.007819 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:09.008241 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:09.008402 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:09.008409 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:09.009377 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:09.009385 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:09.010347 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:09.010358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:09.011052 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:09.011058 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:09.011230 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:09.011952 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:09.011969 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:09.014947 (+ 2978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":255,"threads_started":1}
W20260924 19:01:09.015372 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:09.006897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47985 (local address 127.0.97.62:42585)
0924 19:01:09.007104 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:09.007112 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:09.007416 (+ 304us) server_negotiation.cc:407] Connection header received
0924 19:01:09.007624 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:09.007628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:09.007686 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:09.007790 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:09.008544 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:09.009241 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:09.010487 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:09.010904 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:09.011041 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:09.015106 (+ 4065us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47985: BlockingRecv error: recv got EOF from 127.0.97.2:47985 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 24 19:01:09 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:09 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:09.475641 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:09.467501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.3:39887)
0924 19:01:09.468003 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:09.468023 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:09.468131 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:09.468455 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:09.468458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:09.468833 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:09.469016 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:09.469025 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:09.470050 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:09.470053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:09.470933 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:09.470944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:09.471755 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:09.471762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:09.471968 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:09.472679 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:09.472695 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:09.475404 (+ 2709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":280,"threads_started":1}
W20260924 19:01:09.475814 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:09.467549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39887 (local address 127.0.97.62:42585)
0924 19:01:09.467760 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:09.467766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:09.468080 (+ 314us) server_negotiation.cc:407] Connection header received
0924 19:01:09.468230 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:09.468234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:09.468290 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:09.468410 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:09.469145 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:09.469934 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:09.471059 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:09.471595 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:09.471722 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:09.475591 (+ 3869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:39887: BlockingRecv error: recv got EOF from 127.0.97.3:39887 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 24 19:01:09 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:09 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:09.579491 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:09.571045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.1:60269)
0924 19:01:09.571576 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:09.571591 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:09.571752 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:09.572090 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:09.572096 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:09.572500 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:09.572653 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:09.572661 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:09.573760 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:09.573766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:09.574683 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:09.574693 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:09.575474 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:09.575480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:09.575651 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:09.576277 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:09.576303 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:09.579155 (+ 2852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":299,"threads_started":1}
W20260924 19:01:09.579612 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:09.571093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60269 (local address 127.0.97.62:42585)
0924 19:01:09.571351 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:09.571360 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:09.571679 (+ 319us) server_negotiation.cc:407] Connection header received
0924 19:01:09.571888 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:09.571893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:09.571948 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:09.572049 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:09.572792 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:09.573633 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:09.574834 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:09.575318 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:09.575452 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:09.579421 (+ 3969us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:60269: BlockingRecv error: recv got EOF from 127.0.97.1:60269 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 24 19:01:10 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:10 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:10.024392 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:10.016346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.2:59085)
0924 19:01:10.016830 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:10.016847 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:10.016968 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:10.017255 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:10.017258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:10.017647 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:10.017794 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:10.017802 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:10.018723 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:10.018727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:10.019772 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:10.019783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:10.020699 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:10.020703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:10.020900 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:10.021540 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:10.021557 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:10.024160 (+ 2603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":268,"threads_started":1}
W20260924 19:01:10.024588 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:10.016493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59085 (local address 127.0.97.62:42585)
0924 19:01:10.016775 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:10.016782 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:10.016933 (+ 151us) server_negotiation.cc:407] Connection header received
0924 19:01:10.017071 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:10.017075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:10.017132 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:10.017212 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:10.017917 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:10.018608 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:10.019895 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:10.020540 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:10.020686 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:10.024346 (+ 3660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59085: BlockingRecv error: recv got EOF from 127.0.97.2:59085 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 24 19:01:10 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:10 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:10.488685 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:10.476474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.3:50691)
0924 19:01:10.477013 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:10.477033 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:10.477141 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:10.477412 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:10.477415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:10.477822 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:10.477971 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:10.477980 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:10.478889 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:10.478892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:10.479770 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:10.479781 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:10.480426 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:10.480430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:10.480551 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:10.485893 (+ 5342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:10.485909 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:10.488468 (+ 2559us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":285,"threads_started":1}
W20260924 19:01:10.488876 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:10.476498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50691 (local address 127.0.97.62:42585)
0924 19:01:10.476672 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:10.476676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:10.477111 (+ 435us) server_negotiation.cc:407] Connection header received
0924 19:01:10.477235 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:10.477239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:10.477296 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:10.477374 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:10.478096 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:10.478775 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:10.479885 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:10.480293 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:10.480394 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:10.488648 (+ 8254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:50691: BlockingRecv error: recv got EOF from 127.0.97.3:50691 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 19:01:10 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:10 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 19:01:10.592868 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:10.580706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40951 (local address 127.0.97.62:42585)
0924 19:01:10.580879 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:10.580887 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:10.581120 (+ 233us) server_negotiation.cc:407] Connection header received
0924 19:01:10.581545 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:10.581551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:10.581641 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:10.581747 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:10.582939 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:10.583998 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:10.585961 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:10.586560 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:10.586646 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:10.592620 (+ 5974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:40951: BlockingRecv error: recv got EOF from 127.0.97.1:40951 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 19:01:10.593387 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:10.580478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.1:40951)
0924 19:01:10.581022 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:10.581043 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:10.581395 (+ 352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:10.581796 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:10.581801 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:10.582339 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:10.582570 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:10.582585 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:10.584341 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:10.584346 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:10.585789 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:10.585803 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:10.587053 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:10.587058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:10.587213 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:10.587849 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:10.587871 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:10.593071 (+ 5200us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":402,"threads_started":1}
Sep 24 19:01:11 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:11 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:11.033404 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:11.025295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.2:55007)
0924 19:01:11.025867 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:11.025882 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:11.025989 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:11.026289 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:11.026293 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:11.026765 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:11.027002 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:11.027013 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:11.028332 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:11.028336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:11.029234 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:11.029243 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:11.030049 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:11.030053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:11.030163 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:11.030699 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:11.030715 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:11.033186 (+ 2471us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":299,"threads_started":1}
W20260924 19:01:11.033661 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:11.025471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55007 (local address 127.0.97.62:42585)
0924 19:01:11.025695 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:11.025700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:11.025954 (+ 254us) server_negotiation.cc:407] Connection header received
0924 19:01:11.026087 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:11.026091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:11.026146 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:11.026241 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:11.027128 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:11.028187 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:11.029383 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:11.029900 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:11.029995 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:11.033347 (+ 3352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55007: BlockingRecv error: recv got EOF from 127.0.97.2:55007 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 24 19:01:11 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:11 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:11.497980 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:11.489688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.3:47369)
0924 19:01:11.490169 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:11.490187 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:11.490338 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:11.490657 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:11.490662 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:11.491057 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:11.491232 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:11.491240 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:11.492228 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:11.492236 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:11.493171 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:11.493182 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:11.493995 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:11.494002 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:11.494161 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:11.494736 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:11.494752 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:11.497656 (+ 2904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":253,"threads_started":1}
W20260924 19:01:11.498040 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:11.489688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47369 (local address 127.0.97.62:42585)
0924 19:01:11.489894 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:11.489903 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:11.490270 (+ 367us) server_negotiation.cc:407] Connection header received
0924 19:01:11.490457 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:11.490463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:11.490528 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:11.490626 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:11.491379 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:11.492083 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:11.493333 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:11.493819 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:11.493959 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:11.497828 (+ 3869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47369: BlockingRecv error: recv got EOF from 127.0.97.3:47369 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 19:01:11 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:11 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:11.602828 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:11.594242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.1:36017)
0924 19:01:11.594744 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:11.594759 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:11.594925 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:11.595267 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:11.595273 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:11.595637 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:11.595804 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:11.595812 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:11.596867 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:11.596873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:11.597829 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:11.597840 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:11.598542 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:11.598547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:11.598727 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:11.599505 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:11.599522 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:11.602528 (+ 3006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":269,"threads_started":1}
W20260924 19:01:11.603071 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:11.594358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36017 (local address 127.0.97.62:42585)
0924 19:01:11.594645 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:11.594651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:11.594854 (+ 203us) server_negotiation.cc:407] Connection header received
0924 19:01:11.595050 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:11.595055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:11.595122 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:11.595224 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:11.595948 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:11.596723 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:11.597971 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:11.598398 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:11.598530 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:11.602769 (+ 4239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36017: BlockingRecv error: recv got EOF from 127.0.97.1:36017 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
Sep 24 19:01:12 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:12 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:12.042477 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:12.034246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.2:41421)
0924 19:01:12.034758 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:12.034774 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:12.034918 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:12.035240 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:12.035245 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:12.035633 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:12.035775 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:12.035782 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:12.036760 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:12.036766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:12.037700 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:12.037710 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:12.038411 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:12.038416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:12.038583 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:12.039155 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:12.039171 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:12.042116 (+ 2945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":246,"threads_started":1}
W20260924 19:01:12.042536 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:12.034341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41421 (local address 127.0.97.62:42585)
0924 19:01:12.034573 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:12.034579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:12.034851 (+ 272us) server_negotiation.cc:407] Connection header received
0924 19:01:12.035036 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:12.035041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:12.035110 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:12.035209 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:12.035920 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:12.036616 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:12.037834 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:12.038270 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:12.038404 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:12.042312 (+ 3908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41421: BlockingRecv error: recv got EOF from 127.0.97.2:41421 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 24 19:01:12 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:12 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:12.507335 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:12.498962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.3:53289)
0924 19:01:12.499425 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:12.499446 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:12.499554 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:12.499921 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:12.499924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:12.500405 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:12.500576 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:12.500585 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:12.502024 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:12.502028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:12.502945 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:12.502956 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:12.503597 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:12.503602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:12.503726 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:12.504295 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:12.504312 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:12.507085 (+ 2773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":250,"threads_started":1}
W20260924 19:01:12.507496 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:12.499090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53289 (local address 127.0.97.62:42585)
0924 19:01:12.499300 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:12.499304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:12.499523 (+ 219us) server_negotiation.cc:407] Connection header received
0924 19:01:12.499675 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:12.499679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:12.499734 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:12.499826 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:12.500737 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:12.501886 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:12.503058 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:12.503419 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:12.503506 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:12.507272 (+ 3766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:53289: BlockingRecv error: recv got EOF from 127.0.97.3:53289 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 24 19:01:12 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:12 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:12.612242 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:12.604034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.1:55877)
0924 19:01:12.604568 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:12.604584 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:12.604691 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:12.604971 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:12.604974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:12.605377 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:12.605611 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:12.605621 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:12.606796 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:12.606801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:12.608118 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:12.608130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:12.608702 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:12.608705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:12.608805 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:12.609306 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:12.609322 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:12.611944 (+ 2622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":287,"threads_started":1}
W20260924 19:01:12.612427 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:12.604098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55877 (local address 127.0.97.62:42585)
0924 19:01:12.604274 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:12.604278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:12.604654 (+ 376us) server_negotiation.cc:407] Connection header received
0924 19:01:12.604789 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:12.604792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:12.604847 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:12.604960 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:12.605727 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:12.606626 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:12.608230 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:12.608592 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:12.608680 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:12.612224 (+ 3544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55877: BlockingRecv error: recv got EOF from 127.0.97.1:55877 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 19:01:13 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:01:13 dist-test-slave-9fn1 krb5kdc[8633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:01:13.051766 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:13.043585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42585 (local address 127.0.97.2:52115)
0924 19:01:13.044132 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:13.044148 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:13.044306 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:13.044651 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:13.044655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:13.045066 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:13.045226 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:13.045235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:13.046292 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:13.046299 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:13.047225 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:13.047235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:13.047962 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:13.047967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:13.048138 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:13.048692 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:13.048711 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:13.051411 (+ 2700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42585: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":291,"threads_started":1}
W20260924 19:01:13.051846 9369 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:13.043647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52115 (local address 127.0.97.62:42585)
0924 19:01:13.043868 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:13.043878 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:13.044237 (+ 359us) server_negotiation.cc:407] Connection header received
0924 19:01:13.044442 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:13.044447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:13.044523 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:13.044621 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:13.045369 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:13.046139 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:13.047365 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:13.047813 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:13.047940 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:13.051596 (+ 3656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52115: BlockingRecv error: recv got EOF from 127.0.97.2:52115 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 19:01:13.338260 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 9220
I20260924 19:01:13.349593 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:13.339253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60462 (local address 127.0.97.1:46465)
0924 19:01:13.339787 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:13.339793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:13.339819 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:01:13.339875 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:13.339882 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:13.340093 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:13.340250 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:13.341604 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:13.342608 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:13.343754 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:13.344280 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:13.344381 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:13.346818 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:01:13.346827 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:01:13.346844 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:01:13.346870 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:01:13.349071 (+ 2201us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 19:01:13.349319 (+ 248us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60462: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":263,"threads_started":1}
W20260924 19:01:13.349730 9681 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60462: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 19:01:13.349704 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 9220: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:46465: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 19:01:13.349907 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 9220
I20260924 19:01:13.366169 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 9371
I20260924 19:01:13.378028 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:13.367096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40974 (local address 127.0.97.2:42357)
0924 19:01:13.367731 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:13.367738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:13.367768 (+ 30us) server_negotiation.cc:407] Connection header received
0924 19:01:13.367826 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:13.367832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:13.368166 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:13.368353 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:13.369781 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:13.370795 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:13.371980 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:13.372513 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:13.372649 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:13.375238 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:01:13.375246 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:01:13.375260 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:01:13.375292 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:01:13.377368 (+ 2076us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 19:01:13.377706 (+ 338us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40974: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":321,"threads_started":1}
W20260924 19:01:13.378042 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 9371: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:42357: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 19:01:13.378137 9682 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40974: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 19:01:13.378181 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 9371
I20260924 19:01:13.391049 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 9503
I20260924 19:01:13.402859 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:13.391916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48590 (local address 127.0.97.3:40031)
0924 19:01:13.392489 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:13.392496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:13.392522 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:01:13.392581 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:13.392587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:13.392806 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:13.392953 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:13.394262 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:13.395264 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:13.396362 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:13.396896 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:13.396990 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:13.399290 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:01:13.399298 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:01:13.399311 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:01:13.399347 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:01:13.402354 (+ 3007us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 19:01:13.402594 (+ 240us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48590: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":326,"threads_started":1}
W20260924 19:01:13.402956 9683 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48590: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 19:01:13.402992 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 9503: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:40031: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 19:01:13.403151 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 9503
I20260924 19:01:13.419283 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 9151
I20260924 19:01:13.420490 9213 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:01:13.532931 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 9151
2026-09-24T19:01:13Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20910 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T19:01:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:01:13Z Disabled control of system clock
I20260924 19:01:13.605321 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36185
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 19:01:13.920607 9688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:01:13.920912 9688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:01:13.921043 9688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:01:13.928465 9688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:01:13.928567 9688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:01:13.928639 9688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:01:13.928750 9688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:01:13.928809 9688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:01:13.938076 9688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36185
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:01:13.939332 9688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:01:13.941037 9688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:01:13.953243 9696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:01:13.953310 9694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:01:13.953547 9693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:01:13.953889 9688 server_base.cc:1034] running on GCE node
I20260924 19:01:13.954628 9688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:01:13.955935 9688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:01:13.957151 9688 hybrid_clock.cc:648] HybridClock initialized: now 1790276473957104 us; error 42 us; skew 500 ppm
I20260924 19:01:13.957581 9688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:01:13.959759 9688 webserver.cc:460] Webserver started at http://127.0.97.62:40675/ using document root <none> and password file <none>
I20260924 19:01:13.960445 9688 fs_manager.cc:359] Metadata directory not provided
I20260924 19:01:13.960569 9688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:01:13.960873 9688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:01:13.963660 9688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "3052b83483d1440fbe7f628d83bb1867"
format_stamp: "Formatted at 2026-09-24 19:01:13 on dist-test-slave-9fn1"
server_key: "ea76f8ac4aca583a9115aedc11e9ee1b"
server_key_iv: "d67e594e12f5e5217e28fc6d0a04ecca"
server_key_version: "encryptionkey@0"
I20260924 19:01:13.964378 9688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "3052b83483d1440fbe7f628d83bb1867"
format_stamp: "Formatted at 2026-09-24 19:01:13 on dist-test-slave-9fn1"
server_key: "ea76f8ac4aca583a9115aedc11e9ee1b"
server_key_iv: "d67e594e12f5e5217e28fc6d0a04ecca"
server_key_version: "encryptionkey@0"
I20260924 19:01:13.969095 9688 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260924 19:01:13.972371 9702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:13.973739 9688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 19:01:13.973910 9688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "3052b83483d1440fbe7f628d83bb1867"
format_stamp: "Formatted at 2026-09-24 19:01:13 on dist-test-slave-9fn1"
server_key: "ea76f8ac4aca583a9115aedc11e9ee1b"
server_key_iv: "d67e594e12f5e5217e28fc6d0a04ecca"
server_key_version: "encryptionkey@0"
I20260924 19:01:13.974117 9688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:01:14.014690 9688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:01:14.015619 9688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:01:14.015918 9688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:01:14.033766 9688 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36185
I20260924 19:01:14.033787 9753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36185 every 8 connection(s)
I20260924 19:01:14.035095 9688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:01:14.038825 9754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:01:14.044793 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 9688
I20260924 19:01:14.044978 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:01:14.045781 388 external_mini_cluster.cc:1428] Setting key c05cd28660e07210bb3f84f63bc3c431
I20260924 19:01:14.050050 9754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3052b83483d1440fbe7f628d83bb1867: Bootstrap starting.
I20260924 19:01:14.053035 9754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3052b83483d1440fbe7f628d83bb1867: Neither blocks nor log segments found. Creating new log.
I20260924 19:01:14.054098 9754 log.cc:824] T 00000000000000000000000000000000 P 3052b83483d1440fbe7f628d83bb1867: Log is configured to *not* fsync() on all Append() calls
I20260924 19:01:14.057241 9754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3052b83483d1440fbe7f628d83bb1867: No bootstrap required, opened a new log
I20260924 19:01:14.059723 9757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:14.047960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36208 (local address 127.0.97.62:36185)
0924 19:01:14.048924 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:14.048940 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:14.048979 (+ 39us) server_negotiation.cc:407] Connection header received
0924 19:01:14.050714 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:14.050734 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:14.051160 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:14.051547 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:14.052087 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:14.053385 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:14.054881 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:14.055754 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:14.055929 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:14.057000 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:01:14.057010 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:01:14.057027 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:01:14.057760 (+ 733us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:01:14.057860 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:01:14.057887 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:01:14.058008 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 19:01:14.058179 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 19:01:14.058496 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":341,"threads_started":1}
I20260924 19:01:14.065647 9754 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3052b83483d1440fbe7f628d83bb1867 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3052b83483d1440fbe7f628d83bb1867" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36185 } }
I20260924 19:01:14.065932 9754 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3052b83483d1440fbe7f628d83bb1867 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:01:14.066000 9754 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3052b83483d1440fbe7f628d83bb1867 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3052b83483d1440fbe7f628d83bb1867, State: Initialized, Role: FOLLOWER
I20260924 19:01:14.066434 9754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3052b83483d1440fbe7f628d83bb1867 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3052b83483d1440fbe7f628d83bb1867" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36185 } }
I20260924 19:01:14.066618 9754 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3052b83483d1440fbe7f628d83bb1867 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:01:14.066708 9754 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3052b83483d1440fbe7f628d83bb1867 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:01:14.066876 9754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3052b83483d1440fbe7f628d83bb1867 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:01:14.069132 9754 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3052b83483d1440fbe7f628d83bb1867 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3052b83483d1440fbe7f628d83bb1867" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36185 } }
I20260924 19:01:14.069667 9754 leader_election.cc:302] T 00000000000000000000000000000000 P 3052b83483d1440fbe7f628d83bb1867 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3052b83483d1440fbe7f628d83bb1867; no voters:
I20260924 19:01:14.070190 9754 leader_election.cc:288] T 00000000000000000000000000000000 P 3052b83483d1440fbe7f628d83bb1867 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:01:14.070328 9759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3052b83483d1440fbe7f628d83bb1867 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:01:14.071004 9759 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3052b83483d1440fbe7f628d83bb1867 [term 1 LEADER]: Becoming Leader. State: Replica: 3052b83483d1440fbe7f628d83bb1867, State: Running, Role: LEADER
I20260924 19:01:14.071470 9759 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3052b83483d1440fbe7f628d83bb1867 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3052b83483d1440fbe7f628d83bb1867" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36185 } }
I20260924 19:01:14.072124 9754 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3052b83483d1440fbe7f628d83bb1867 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:01:14.074172 9760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3052b83483d1440fbe7f628d83bb1867 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3052b83483d1440fbe7f628d83bb1867" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3052b83483d1440fbe7f628d83bb1867" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36185 } } }
I20260924 19:01:14.074647 9760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3052b83483d1440fbe7f628d83bb1867 [sys.catalog]: This master's current role is: LEADER
I20260924 19:01:14.074538 9761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3052b83483d1440fbe7f628d83bb1867 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3052b83483d1440fbe7f628d83bb1867. Latest consensus state: current_term: 1 leader_uuid: "3052b83483d1440fbe7f628d83bb1867" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3052b83483d1440fbe7f628d83bb1867" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36185 } } }
I20260924 19:01:14.074846 9761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3052b83483d1440fbe7f628d83bb1867 [sys.catalog]: This master's current role is: LEADER
I20260924 19:01:14.075533 9765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:01:14.082688 9765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:01:14.090399 9765 catalog_manager.cc:1347] Generated new cluster ID: d825ce13079548e18578cefb1513ab9d
I20260924 19:01:14.090515 9765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:01:14.121621 9765 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:01:14.122938 9765 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:01:14.128433 9765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3052b83483d1440fbe7f628d83bb1867: Generated new TSK 0
I20260924 19:01:14.129205 9765 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:01:14.144773 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36185
--builtin_ntp_servers=127.0.97.20:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 19:01:14.462823 9778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:01:14.463083 9778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:01:14.463244 9778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:01:14.470333 9778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:01:14.470419 9778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:01:14.470585 9778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:01:14.480026 9778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:01:14.481209 9778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:01:14.483011 9778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:01:14.495425 9786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:01:14.495497 9784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:01:14.495532 9783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:01:14.496466 9778 server_base.cc:1034] running on GCE node
I20260924 19:01:14.497093 9778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:01:14.498250 9778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:01:14.499492 9778 hybrid_clock.cc:648] HybridClock initialized: now 1790276474499439 us; error 47 us; skew 500 ppm
I20260924 19:01:14.499804 9778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:01:14.501994 9778 webserver.cc:460] Webserver started at http://127.0.97.1:44713/ using document root <none> and password file <none>
I20260924 19:01:14.502681 9778 fs_manager.cc:359] Metadata directory not provided
I20260924 19:01:14.502792 9778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:01:14.503113 9778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:01:14.505915 9778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/data/instance:
uuid: "a9655cf66b1a45c7b0c4fab3d2417832"
format_stamp: "Formatted at 2026-09-24 19:01:14 on dist-test-slave-9fn1"
server_key: "eea1b8b35415c7ad7c03c6cce9a7b2d2"
server_key_iv: "f3ac94c7f265159db0055f9b42452910"
server_key_version: "encryptionkey@0"
I20260924 19:01:14.506682 9778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a9655cf66b1a45c7b0c4fab3d2417832"
format_stamp: "Formatted at 2026-09-24 19:01:14 on dist-test-slave-9fn1"
server_key: "eea1b8b35415c7ad7c03c6cce9a7b2d2"
server_key_iv: "f3ac94c7f265159db0055f9b42452910"
server_key_version: "encryptionkey@0"
I20260924 19:01:14.511508 9778 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 19:01:14.514705 9792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:14.516021 9778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:01:14.516202 9778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "a9655cf66b1a45c7b0c4fab3d2417832"
format_stamp: "Formatted at 2026-09-24 19:01:14 on dist-test-slave-9fn1"
server_key: "eea1b8b35415c7ad7c03c6cce9a7b2d2"
server_key_iv: "f3ac94c7f265159db0055f9b42452910"
server_key_version: "encryptionkey@0"
I20260924 19:01:14.516418 9778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:01:14.546859 9778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:01:14.547734 9778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:01:14.548026 9778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:01:14.548947 9778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:01:14.550379 9778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:01:14.550499 9778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:14.550634 9778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:01:14.550722 9778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:14.587122 9778 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33251
I20260924 19:01:14.587224 9904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33251 every 8 connection(s)
I20260924 19:01:14.588886 9778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:01:14.596984 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 9778
I20260924 19:01:14.597214 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/wal/instance
I20260924 19:01:14.597914 388 external_mini_cluster.cc:1428] Setting key c48b92997e3fed875629ece6c38d98f8
I20260924 19:01:14.600582 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36185
--builtin_ntp_servers=127.0.97.20:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 19:01:14.601558 9907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:14.591632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53825 (local address 127.0.97.62:36185)
0924 19:01:14.592060 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:14.592065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:14.593240 (+ 1175us) server_negotiation.cc:407] Connection header received
0924 19:01:14.594914 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:14.594918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:14.594981 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:14.595080 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:14.596046 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:14.596965 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:14.598800 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:14.599215 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:14.599352 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:14.600715 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:01:14.600723 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:01:14.600728 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:01:14.600747 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:01:14.600798 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:01:14.600813 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:01:14.600917 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 19:01:14.601119 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 19:01:14.601203 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":205,"threads_started":1}
I20260924 19:01:14.602480 9908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:14.592262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36185 (local address 127.0.97.1:53825)
0924 19:01:14.593095 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:14.593134 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:14.594693 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:14.595263 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:14.595270 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:14.595349 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:14.595906 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:14.595922 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:14.597142 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:14.597148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:14.598644 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:14.598659 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:14.599478 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:14.599486 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:14.599704 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:14.600408 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:01:14.600410 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:14.600446 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:01:14.600514 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:01:14.600915 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:01:14.600926 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:01:14.600936 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 19:01:14.601111 (+ 175us) client_negotiation.cc:240] Negotiation successful
0924 19:01:14.601474 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":268,"threads_started":1}
I20260924 19:01:14.603961 9905 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36185
I20260924 19:01:14.604270 9905 heartbeater.cc:464] Registering TS with master...
I20260924 19:01:14.605005 9905 heartbeater.cc:511] Master 127.0.97.62:36185 requested a full tablet report, sending...
I20260924 19:01:14.607290 9719 ts_manager.cc:194] Registered new tserver with Master: a9655cf66b1a45c7b0c4fab3d2417832 (127.0.97.1:33251)
I20260924 19:01:14.608702 9719 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:53825
W20260924 19:01:14.916818 9909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:01:14.917109 9909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:01:14.917266 9909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:01:14.924816 9909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:01:14.924913 9909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:01:14.925137 9909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:01:14.934468 9909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:01:14.935678 9909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:01:14.937366 9909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:01:14.950475 9914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:01:14.950433 9915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:01:14.950520 9917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:01:14.951442 9909 server_base.cc:1034] running on GCE node
I20260924 19:01:14.952018 9909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:01:14.952917 9909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:01:14.954157 9909 hybrid_clock.cc:648] HybridClock initialized: now 1790276474954114 us; error 56 us; skew 500 ppm
I20260924 19:01:14.954452 9909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:01:14.956591 9909 webserver.cc:460] Webserver started at http://127.0.97.2:42293/ using document root <none> and password file <none>
I20260924 19:01:14.957254 9909 fs_manager.cc:359] Metadata directory not provided
I20260924 19:01:14.957386 9909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:01:14.957742 9909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:01:14.960356 9909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/data/instance:
uuid: "5fe33d76b9e847f2b24fe00ac4e4ba16"
format_stamp: "Formatted at 2026-09-24 19:01:14 on dist-test-slave-9fn1"
server_key: "d0e5c0ab7a3ecc09915989b1719125a8"
server_key_iv: "1d34b075ab673b7c9e1a4a1cfb4611ff"
server_key_version: "encryptionkey@0"
I20260924 19:01:14.961150 9909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5fe33d76b9e847f2b24fe00ac4e4ba16"
format_stamp: "Formatted at 2026-09-24 19:01:14 on dist-test-slave-9fn1"
server_key: "d0e5c0ab7a3ecc09915989b1719125a8"
server_key_iv: "1d34b075ab673b7c9e1a4a1cfb4611ff"
server_key_version: "encryptionkey@0"
I20260924 19:01:14.965945 9909 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 19:01:14.969066 9923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:14.970352 9909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:01:14.970535 9909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "5fe33d76b9e847f2b24fe00ac4e4ba16"
format_stamp: "Formatted at 2026-09-24 19:01:14 on dist-test-slave-9fn1"
server_key: "d0e5c0ab7a3ecc09915989b1719125a8"
server_key_iv: "1d34b075ab673b7c9e1a4a1cfb4611ff"
server_key_version: "encryptionkey@0"
I20260924 19:01:14.970742 9909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:01:14.984229 9909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:01:14.985042 9909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:01:14.985313 9909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:01:14.986289 9909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:01:14.987697 9909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:01:14.987823 9909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:14.987944 9909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:01:14.988034 9909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:15.020072 9909 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45293
I20260924 19:01:15.020124 10035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45293 every 8 connection(s)
I20260924 19:01:15.021373 9909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:01:15.022763 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 9909
I20260924 19:01:15.022934 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/wal/instance
I20260924 19:01:15.023388 388 external_mini_cluster.cc:1428] Setting key facfea815014e623bb73a39b5bbb0f82
I20260924 19:01:15.026324 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36185
--builtin_ntp_servers=127.0.97.20:39653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 19:01:15.035296 9907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:15.025303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35409 (local address 127.0.97.62:36185)
0924 19:01:15.025493 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:15.025556 (+ 63us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:15.027181 (+ 1625us) server_negotiation.cc:407] Connection header received
0924 19:01:15.028903 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:15.028908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:15.028997 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:15.029114 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:15.030217 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:15.030928 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:15.032448 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:15.032864 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:15.033003 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:15.034437 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:01:15.034442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:01:15.034447 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:01:15.034475 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:01:15.034506 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:01:15.034517 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:01:15.034611 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 19:01:15.034836 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 19:01:15.034972 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 19:01:15.036335 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:15.026024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36185 (local address 127.0.97.2:35409)
0924 19:01:15.027039 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:15.027080 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:15.028628 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:15.029335 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:15.029343 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:15.029435 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:15.030056 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:15.030073 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:15.031084 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:15.031091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:15.032305 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:15.032316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:15.033087 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:15.033094 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:15.033391 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:15.034142 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:01:15.034145 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:15.034185 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:01:15.034255 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:01:15.034611 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:01:15.034624 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:01:15.034635 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 19:01:15.034820 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 19:01:15.035209 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":375,"threads_started":1}
I20260924 19:01:15.041612 10036 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36185
I20260924 19:01:15.041915 10036 heartbeater.cc:464] Registering TS with master...
I20260924 19:01:15.042654 10036 heartbeater.cc:511] Master 127.0.97.62:36185 requested a full tablet report, sending...
I20260924 19:01:15.044160 9719 ts_manager.cc:194] Registered new tserver with Master: 5fe33d76b9e847f2b24fe00ac4e4ba16 (127.0.97.2:45293)
I20260924 19:01:15.044955 9719 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:35409
W20260924 19:01:15.348613 10039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:01:15.348850 10039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:01:15.348925 10039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:01:15.355851 10039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:01:15.355958 10039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:01:15.356051 10039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:01:15.364588 10039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36185
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:01:15.365717 10039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:01:15.367293 10039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:01:15.380000 10044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:01:15.380038 10045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:01:15.380090 10047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:01:15.380653 10039 server_base.cc:1034] running on GCE node
I20260924 19:01:15.381268 10039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:01:15.382213 10039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:01:15.383438 10039 hybrid_clock.cc:648] HybridClock initialized: now 1790276475383395 us; error 35 us; skew 500 ppm
I20260924 19:01:15.383782 10039 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:01:15.386107 10039 webserver.cc:460] Webserver started at http://127.0.97.3:34305/ using document root <none> and password file <none>
I20260924 19:01:15.386754 10039 fs_manager.cc:359] Metadata directory not provided
I20260924 19:01:15.386891 10039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:01:15.387207 10039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:01:15.389899 10039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/data/instance:
uuid: "f10e831814b1461891f8db3d972e03b1"
format_stamp: "Formatted at 2026-09-24 19:01:15 on dist-test-slave-9fn1"
server_key: "d696bc81535916e57bdb45c3f5ef1994"
server_key_iv: "d453587b3fb5e98de86eed8a8ba08324"
server_key_version: "encryptionkey@0"
I20260924 19:01:15.390640 10039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f10e831814b1461891f8db3d972e03b1"
format_stamp: "Formatted at 2026-09-24 19:01:15 on dist-test-slave-9fn1"
server_key: "d696bc81535916e57bdb45c3f5ef1994"
server_key_iv: "d453587b3fb5e98de86eed8a8ba08324"
server_key_version: "encryptionkey@0"
I20260924 19:01:15.395987 10039 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:01:15.399195 10053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:15.400314 10039 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 19:01:15.400481 10039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "f10e831814b1461891f8db3d972e03b1"
format_stamp: "Formatted at 2026-09-24 19:01:15 on dist-test-slave-9fn1"
server_key: "d696bc81535916e57bdb45c3f5ef1994"
server_key_iv: "d453587b3fb5e98de86eed8a8ba08324"
server_key_version: "encryptionkey@0"
I20260924 19:01:15.400682 10039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:01:15.429214 10039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:01:15.430104 10039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:01:15.430380 10039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:01:15.431341 10039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:01:15.432760 10039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:01:15.432912 10039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:15.433058 10039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:01:15.433148 10039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:15.464943 10039 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39591
I20260924 19:01:15.464983 10165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39591 every 8 connection(s)
I20260924 19:01:15.466248 10039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:01:15.470054 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 10039
I20260924 19:01:15.470204 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/wal/instance
I20260924 19:01:15.470758 388 external_mini_cluster.cc:1428] Setting key fcbc96ab79733ccf51f16fe9dfc533be
I20260924 19:01:15.479696 9907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:15.468754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39561 (local address 127.0.97.62:36185)
0924 19:01:15.468927 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:15.468931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:15.470533 (+ 1602us) server_negotiation.cc:407] Connection header received
0924 19:01:15.472755 (+ 2222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:15.472760 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:15.472851 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:15.472960 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:15.474557 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:15.475381 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:15.476938 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:15.477477 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:15.477635 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:15.479083 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:01:15.479088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:01:15.479091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:01:15.479107 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:01:15.479136 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:01:15.479147 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:01:15.479215 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 19:01:15.479398 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 19:01:15.479466 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 19:01:15.480633 10168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:15.469258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36185 (local address 127.0.97.3:39561)
0924 19:01:15.470353 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:15.470400 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:15.472465 (+ 2065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:15.473204 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:15.473217 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:15.473318 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:15.474366 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:15.474394 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:15.475510 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:15.475514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:15.476793 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:15.476805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:15.477747 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:15.477753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:15.478041 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:15.478716 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:01:15.478718 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:15.478801 (+ 83us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:01:15.478909 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:01:15.479227 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:01:15.479234 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:01:15.479239 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:01:15.479379 (+ 140us) client_negotiation.cc:240] Negotiation successful
0924 19:01:15.479638 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":303,"threads_started":1}
I20260924 19:01:15.481894 10166 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36185
I20260924 19:01:15.482168 10166 heartbeater.cc:464] Registering TS with master...
I20260924 19:01:15.482834 10166 heartbeater.cc:511] Master 127.0.97.62:36185 requested a full tablet report, sending...
I20260924 19:01:15.484138 9719 ts_manager.cc:194] Registered new tserver with Master: f10e831814b1461891f8db3d972e03b1 (127.0.97.3:39591)
I20260924 19:01:15.485018 9719 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:39561
I20260924 19:01:15.486680 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 19:01:15.492458 9907 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:15.490866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46648 (local address 127.0.97.62:36185)
0924 19:01:15.491072 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:15.491080 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:15.491443 (+ 363us) server_negotiation.cc:407] Connection header received
0924 19:01:15.491622 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:15.491627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:15.491705 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:15.491811 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:15.492214 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46648: BlockingRecv error: recv got EOF from 127.0.0.1:46648 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 19:01:15.493325 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 9778
I20260924 19:01:15.506807 10175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:15.494309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47002 (local address 127.0.97.1:33251)
0924 19:01:15.494801 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:15.494808 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:15.494831 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:01:15.494886 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:15.494892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:15.495073 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:15.495192 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:15.495501 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:15.496440 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:15.497604 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:15.498115 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:15.498213 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:15.505156 (+ 6943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:01:15.505165 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:01:15.505178 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:01:15.506005 (+ 827us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:01:15.506065 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:01:15.506084 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:01:15.506172 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 19:01:15.506318 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 19:01:15.506457 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":272,"threads_started":1}
I20260924 19:01:15.508019 9900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:01:15.591566 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 9778
I20260924 19:01:15.608598 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 9909
I20260924 19:01:15.616044 10177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:15.609564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47720 (local address 127.0.97.2:45293)
0924 19:01:15.610095 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:15.610103 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:15.610128 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:01:15.610185 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:15.610191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:15.610424 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:15.610549 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:15.610888 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:15.611851 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:15.612975 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:15.613571 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:15.613675 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:15.614705 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:01:15.614712 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:01:15.614724 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:01:15.615288 (+ 564us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:01:15.615329 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:01:15.615347 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:01:15.615437 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 19:01:15.615567 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 19:01:15.615705 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":275,"threads_started":1}
I20260924 19:01:15.617336 10031 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:01:15.698638 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 9909
I20260924 19:01:15.715034 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 10039
I20260924 19:01:15.723132 10179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:15.715981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41296 (local address 127.0.97.3:39591)
0924 19:01:15.716565 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:15.716572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:15.716601 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:01:15.716659 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:15.716665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:15.716891 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:15.717067 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:15.717538 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:15.718651 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:15.719932 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:15.720480 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:15.720625 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:15.721765 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:01:15.721773 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:01:15.721786 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:01:15.722346 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:01:15.722387 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:01:15.722407 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:01:15.722542 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 19:01:15.722651 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 19:01:15.722776 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":310,"threads_started":1}
I20260924 19:01:15.724305 10161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:01:15.805922 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 10039
I20260924 19:01:15.821091 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 9688
I20260924 19:01:15.822319 9749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:01:15.922611 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 9688
2026-09-24T19:01:15Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2384 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T19:01:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:01:15Z Disabled control of system clock
I20260924 19:01:15.990588 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39897
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:37709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 19:01:16.313201 10185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:01:16.313481 10185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:01:16.313674 10185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:01:16.320924 10185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:01:16.321020 10185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:01:16.321133 10185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:01:16.321244 10185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:01:16.321331 10185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:01:16.330662 10185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39897
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:01:16.332002 10185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:01:16.333828 10185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:01:16.345265 10193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:01:16.345276 10190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:01:16.345247 10191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:01:16.346276 10185 server_base.cc:1034] running on GCE node
I20260924 19:01:16.347081 10185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:01:16.348423 10185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:01:16.349645 10185 hybrid_clock.cc:648] HybridClock initialized: now 1790276476349598 us; error 56 us; skew 500 ppm
I20260924 19:01:16.349990 10185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:01:16.352207 10185 webserver.cc:460] Webserver started at http://127.0.97.62:35325/ using document root <none> and password file <none>
I20260924 19:01:16.352895 10185 fs_manager.cc:359] Metadata directory not provided
I20260924 19:01:16.353029 10185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:01:16.353372 10185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:01:16.356232 10185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "75f0ebf32d264855a06b238847de49f3"
format_stamp: "Formatted at 2026-09-24 19:01:16 on dist-test-slave-9fn1"
server_key: "29892a2045c0a00a1f5a2467e3aa0111"
server_key_iv: "42525c5ee7290cb4e3eb13f334d3f1c5"
server_key_version: "encryptionkey@0"
I20260924 19:01:16.357004 10185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "75f0ebf32d264855a06b238847de49f3"
format_stamp: "Formatted at 2026-09-24 19:01:16 on dist-test-slave-9fn1"
server_key: "29892a2045c0a00a1f5a2467e3aa0111"
server_key_iv: "42525c5ee7290cb4e3eb13f334d3f1c5"
server_key_version: "encryptionkey@0"
I20260924 19:01:16.361850 10185 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 19:01:16.365377 10199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:16.366649 10185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:01:16.366834 10185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "75f0ebf32d264855a06b238847de49f3"
format_stamp: "Formatted at 2026-09-24 19:01:16 on dist-test-slave-9fn1"
server_key: "29892a2045c0a00a1f5a2467e3aa0111"
server_key_iv: "42525c5ee7290cb4e3eb13f334d3f1c5"
server_key_version: "encryptionkey@0"
I20260924 19:01:16.367044 10185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:01:16.372579 10185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:01:16.373317 10185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:01:16.373703 10185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:01:16.391278 10185 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39897
I20260924 19:01:16.391367 10250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39897 every 8 connection(s)
I20260924 19:01:16.392664 10185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:01:16.396167 10251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:01:16.399732 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 10185
I20260924 19:01:16.399940 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:01:16.400549 388 external_mini_cluster.cc:1428] Setting key 03a3000a6fea8a2035700e4dc9802b3b
I20260924 19:01:16.407815 10251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75f0ebf32d264855a06b238847de49f3: Bootstrap starting.
I20260924 19:01:16.409786 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:16.402511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34758 (local address 127.0.97.62:39897)
0924 19:01:16.403529 (+ 1018us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:16.403542 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:16.403593 (+ 51us) server_negotiation.cc:407] Connection header received
0924 19:01:16.406041 (+ 2448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:16.406069 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:16.406542 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:16.406920 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:16.407875 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:01:16.407884 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:01:16.407900 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:01:16.407918 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:01:16.407988 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:01:16.408008 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:01:16.408220 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 19:01:16.408343 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 19:01:16.408573 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":287,"threads_started":1}
I20260924 19:01:16.411170 10251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75f0ebf32d264855a06b238847de49f3: Neither blocks nor log segments found. Creating new log.
I20260924 19:01:16.412146 10251 log.cc:824] T 00000000000000000000000000000000 P 75f0ebf32d264855a06b238847de49f3: Log is configured to *not* fsync() on all Append() calls
I20260924 19:01:16.414706 10251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75f0ebf32d264855a06b238847de49f3: No bootstrap required, opened a new log
I20260924 19:01:16.421774 10251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75f0ebf32d264855a06b238847de49f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f0ebf32d264855a06b238847de49f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } }
I20260924 19:01:16.422104 10251 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75f0ebf32d264855a06b238847de49f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:01:16.422195 10251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75f0ebf32d264855a06b238847de49f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75f0ebf32d264855a06b238847de49f3, State: Initialized, Role: FOLLOWER
I20260924 19:01:16.422745 10251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75f0ebf32d264855a06b238847de49f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f0ebf32d264855a06b238847de49f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } }
I20260924 19:01:16.422982 10251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75f0ebf32d264855a06b238847de49f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:01:16.423076 10251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75f0ebf32d264855a06b238847de49f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:01:16.423189 10251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75f0ebf32d264855a06b238847de49f3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:01:16.425601 10251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75f0ebf32d264855a06b238847de49f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f0ebf32d264855a06b238847de49f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } }
I20260924 19:01:16.426021 10251 leader_election.cc:302] T 00000000000000000000000000000000 P 75f0ebf32d264855a06b238847de49f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75f0ebf32d264855a06b238847de49f3; no voters:
I20260924 19:01:16.426542 10251 leader_election.cc:288] T 00000000000000000000000000000000 P 75f0ebf32d264855a06b238847de49f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:01:16.426692 10256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75f0ebf32d264855a06b238847de49f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:01:16.427385 10256 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75f0ebf32d264855a06b238847de49f3 [term 1 LEADER]: Becoming Leader. State: Replica: 75f0ebf32d264855a06b238847de49f3, State: Running, Role: LEADER
I20260924 19:01:16.427942 10256 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75f0ebf32d264855a06b238847de49f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f0ebf32d264855a06b238847de49f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } }
I20260924 19:01:16.428521 10251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75f0ebf32d264855a06b238847de49f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:01:16.430682 10257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75f0ebf32d264855a06b238847de49f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75f0ebf32d264855a06b238847de49f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f0ebf32d264855a06b238847de49f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } } }
I20260924 19:01:16.431166 10257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75f0ebf32d264855a06b238847de49f3 [sys.catalog]: This master's current role is: LEADER
I20260924 19:01:16.430995 10258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75f0ebf32d264855a06b238847de49f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75f0ebf32d264855a06b238847de49f3. Latest consensus state: current_term: 1 leader_uuid: "75f0ebf32d264855a06b238847de49f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75f0ebf32d264855a06b238847de49f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39897 } } }
I20260924 19:01:16.431352 10258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75f0ebf32d264855a06b238847de49f3 [sys.catalog]: This master's current role is: LEADER
I20260924 19:01:16.432101 10264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:01:16.442874 10264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:01:16.451622 10264 catalog_manager.cc:1347] Generated new cluster ID: d8f7ff892bea4816b61c7df750ff7db4
I20260924 19:01:16.451756 10264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:01:16.459915 10264 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:01:16.460289 10264 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:01:16.470489 10264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75f0ebf32d264855a06b238847de49f3: Generated new TSK 0
I20260924 19:01:16.471359 10264 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:01:16.480813 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39897
--builtin_ntp_servers=127.0.97.20:37709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 19:01:16.805862 10275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:01:16.806164 10275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:01:16.806344 10275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:01:16.813542 10275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:01:16.813690 10275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:01:16.813871 10275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:01:16.823609 10275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:01:16.824985 10275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:01:16.826810 10275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:01:16.840049 10280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:01:16.840026 10283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:01:16.839996 10281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:01:16.840916 10275 server_base.cc:1034] running on GCE node
I20260924 19:01:16.841557 10275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:01:16.842465 10275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:01:16.843698 10275 hybrid_clock.cc:648] HybridClock initialized: now 1790276476843626 us; error 54 us; skew 500 ppm
I20260924 19:01:16.844022 10275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:01:16.846202 10275 webserver.cc:460] Webserver started at http://127.0.97.1:43539/ using document root <none> and password file <none>
I20260924 19:01:16.846884 10275 fs_manager.cc:359] Metadata directory not provided
I20260924 19:01:16.847010 10275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:01:16.847316 10275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:01:16.850252 10275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/data/instance:
uuid: "43dcc13329a548078770d63c4bb19b9c"
format_stamp: "Formatted at 2026-09-24 19:01:16 on dist-test-slave-9fn1"
server_key: "6f7e6215d73081b288386acc3c44613a"
server_key_iv: "b0b9559f23336cb26e0af6010f71fcea"
server_key_version: "encryptionkey@0"
I20260924 19:01:16.851068 10275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/wal/instance:
uuid: "43dcc13329a548078770d63c4bb19b9c"
format_stamp: "Formatted at 2026-09-24 19:01:16 on dist-test-slave-9fn1"
server_key: "6f7e6215d73081b288386acc3c44613a"
server_key_iv: "b0b9559f23336cb26e0af6010f71fcea"
server_key_version: "encryptionkey@0"
I20260924 19:01:16.856597 10275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260924 19:01:16.860075 10289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:16.861440 10275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:01:16.861687 10275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "43dcc13329a548078770d63c4bb19b9c"
format_stamp: "Formatted at 2026-09-24 19:01:16 on dist-test-slave-9fn1"
server_key: "6f7e6215d73081b288386acc3c44613a"
server_key_iv: "b0b9559f23336cb26e0af6010f71fcea"
server_key_version: "encryptionkey@0"
I20260924 19:01:16.861943 10275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:01:16.868476 10275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:01:16.869328 10275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:01:16.869678 10275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:01:16.870656 10275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:01:16.872054 10275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:01:16.872179 10275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:16.872308 10275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:01:16.872393 10275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:16.904596 10275 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45381
I20260924 19:01:16.904646 10401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45381 every 8 connection(s)
I20260924 19:01:16.905961 10275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:01:16.913705 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:16.908410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51791 (local address 127.0.97.62:39897)
0924 19:01:16.908589 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:16.908594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:16.910058 (+ 1464us) server_negotiation.cc:407] Connection header received
0924 19:01:16.911666 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:16.911671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:16.911720 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:16.911814 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:16.912981 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:01:16.912986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:01:16.912989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:01:16.913003 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:01:16.913045 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:01:16.913059 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:01:16.913151 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 19:01:16.913363 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 19:01:16.913422 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 19:01:16.914458 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 10275
I20260924 19:01:16.914502 10404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:16.909006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39897 (local address 127.0.97.1:51791)
0924 19:01:16.909905 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:16.909939 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:16.911394 (+ 1455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:16.911999 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:16.912006 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:16.912118 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:16.912722 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:01:16.912724 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:16.912758 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:01:16.912836 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:01:16.913155 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:01:16.913162 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:01:16.913167 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:01:16.913366 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 19:01:16.913603 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":244,"threads_started":1}
I20260924 19:01:16.914652 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-0/wal/instance
I20260924 19:01:16.915236 388 external_mini_cluster.cc:1428] Setting key 4554483ffd1aab98a21240e6166e4b10
I20260924 19:01:16.916095 10402 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39897
I20260924 19:01:16.916412 10402 heartbeater.cc:464] Registering TS with master...
I20260924 19:01:16.917035 10402 heartbeater.cc:511] Master 127.0.97.62:39897 requested a full tablet report, sending...
I20260924 19:01:16.918169 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39897
--builtin_ntp_servers=127.0.97.20:37709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 19:01:16.919087 10216 ts_manager.cc:194] Registered new tserver with Master: 43dcc13329a548078770d63c4bb19b9c (127.0.97.1:45381)
W20260924 19:01:17.242179 10405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:01:17.242443 10405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:01:17.242578 10405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:01:17.249723 10405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:01:17.249810 10405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:01:17.249995 10405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:01:17.258823 10405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:01:17.260022 10405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:01:17.261696 10405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:01:17.273932 10410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:01:17.274027 10411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:01:17.274320 10413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:01:17.274961 10405 server_base.cc:1034] running on GCE node
I20260924 19:01:17.275552 10405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:01:17.276372 10405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:01:17.277593 10405 hybrid_clock.cc:648] HybridClock initialized: now 1790276477277524 us; error 50 us; skew 500 ppm
I20260924 19:01:17.277894 10405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:01:17.280000 10405 webserver.cc:460] Webserver started at http://127.0.97.2:41763/ using document root <none> and password file <none>
I20260924 19:01:17.280653 10405 fs_manager.cc:359] Metadata directory not provided
I20260924 19:01:17.280766 10405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:01:17.281121 10405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:01:17.285377 10405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/data/instance:
uuid: "d8930cc816944ab68909bb17e197a3ea"
format_stamp: "Formatted at 2026-09-24 19:01:17 on dist-test-slave-9fn1"
server_key: "ece3cd86f7ec29f3da2a20035ea6eca7"
server_key_iv: "4466097b201092d19bb06e02807db36e"
server_key_version: "encryptionkey@0"
I20260924 19:01:17.286211 10405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d8930cc816944ab68909bb17e197a3ea"
format_stamp: "Formatted at 2026-09-24 19:01:17 on dist-test-slave-9fn1"
server_key: "ece3cd86f7ec29f3da2a20035ea6eca7"
server_key_iv: "4466097b201092d19bb06e02807db36e"
server_key_version: "encryptionkey@0"
I20260924 19:01:17.291178 10405 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:01:17.294507 10419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:17.295804 10405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:01:17.295976 10405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "d8930cc816944ab68909bb17e197a3ea"
format_stamp: "Formatted at 2026-09-24 19:01:17 on dist-test-slave-9fn1"
server_key: "ece3cd86f7ec29f3da2a20035ea6eca7"
server_key_iv: "4466097b201092d19bb06e02807db36e"
server_key_version: "encryptionkey@0"
I20260924 19:01:17.296216 10405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:01:17.301800 10405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:01:17.302534 10405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:01:17.302794 10405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:01:17.303646 10405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:01:17.304924 10405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:01:17.305042 10405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:17.305194 10405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:01:17.305282 10405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:17.336330 10405 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34619
I20260924 19:01:17.336383 10531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34619 every 8 connection(s)
I20260924 19:01:17.337667 10405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:01:17.339180 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 10405
I20260924 19:01:17.339365 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-1/wal/instance
I20260924 19:01:17.339784 388 external_mini_cluster.cc:1428] Setting key c6c9e7acddc603d9f0000a29748cc68d
I20260924 19:01:17.342464 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39897
--builtin_ntp_servers=127.0.97.20:37709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 19:01:17.347144 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:17.341545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56235 (local address 127.0.97.62:39897)
0924 19:01:17.341733 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:17.341741 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:17.343327 (+ 1586us) server_negotiation.cc:407] Connection header received
0924 19:01:17.344898 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:17.344906 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:17.344966 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:17.345069 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:17.346338 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:01:17.346345 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:01:17.346353 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:01:17.346374 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:01:17.346412 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:01:17.346430 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:01:17.346545 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 19:01:17.346723 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 19:01:17.346783 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:01:17.347718 10534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:17.342166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39897 (local address 127.0.97.2:56235)
0924 19:01:17.343155 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:17.343196 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:17.344651 (+ 1455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:17.345256 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:17.345263 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:17.345350 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:17.346037 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:01:17.346040 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:17.346080 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:01:17.346156 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:01:17.346532 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:01:17.346539 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:01:17.346544 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:01:17.346740 (+ 196us) client_negotiation.cc:240] Negotiation successful
0924 19:01:17.346923 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":312,"threads_started":1}
I20260924 19:01:17.349046 10532 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39897
I20260924 19:01:17.349366 10532 heartbeater.cc:464] Registering TS with master...
I20260924 19:01:17.350061 10532 heartbeater.cc:511] Master 127.0.97.62:39897 requested a full tablet report, sending...
I20260924 19:01:17.351397 10216 ts_manager.cc:194] Registered new tserver with Master: d8930cc816944ab68909bb17e197a3ea (127.0.97.2:34619)
W20260924 19:01:17.684576 10535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:01:17.684852 10535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:01:17.685022 10535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:01:17.692441 10535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:01:17.692585 10535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:01:17.692772 10535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:01:17.702111 10535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:01:17.703385 10535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:01:17.705106 10535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:01:17.718168 10540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:01:17.718212 10541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:01:17.718235 10543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:01:17.718926 10535 server_base.cc:1034] running on GCE node
I20260924 19:01:17.719627 10535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:01:17.720538 10535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:01:17.721727 10535 hybrid_clock.cc:648] HybridClock initialized: now 1790276477721665 us; error 55 us; skew 500 ppm
I20260924 19:01:17.722088 10535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:01:17.724272 10535 webserver.cc:460] Webserver started at http://127.0.97.3:35283/ using document root <none> and password file <none>
I20260924 19:01:17.724943 10535 fs_manager.cc:359] Metadata directory not provided
I20260924 19:01:17.725167 10535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:01:17.725600 10535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:01:17.728385 10535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/data/instance:
uuid: "a124b66f0f264af5adeaae72769d9f6d"
format_stamp: "Formatted at 2026-09-24 19:01:17 on dist-test-slave-9fn1"
server_key: "75da23845d3537a7c9cb3d70aad600a1"
server_key_iv: "ce7bc11383d15eded5b8081244b616d6"
server_key_version: "encryptionkey@0"
I20260924 19:01:17.729135 10535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a124b66f0f264af5adeaae72769d9f6d"
format_stamp: "Formatted at 2026-09-24 19:01:17 on dist-test-slave-9fn1"
server_key: "75da23845d3537a7c9cb3d70aad600a1"
server_key_iv: "ce7bc11383d15eded5b8081244b616d6"
server_key_version: "encryptionkey@0"
I20260924 19:01:17.734457 10535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 19:01:17.737831 10549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:17.739295 10535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:01:17.739468 10535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "a124b66f0f264af5adeaae72769d9f6d"
format_stamp: "Formatted at 2026-09-24 19:01:17 on dist-test-slave-9fn1"
server_key: "75da23845d3537a7c9cb3d70aad600a1"
server_key_iv: "ce7bc11383d15eded5b8081244b616d6"
server_key_version: "encryptionkey@0"
I20260924 19:01:17.739682 10535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:01:17.745322 10535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:01:17.746079 10535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:01:17.746333 10535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:01:17.747200 10535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:01:17.748533 10535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:01:17.748641 10535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:17.748773 10535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:01:17.748867 10535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:17.781517 10535 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41531
I20260924 19:01:17.781615 10661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41531 every 8 connection(s)
I20260924 19:01:17.782853 10535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:01:17.784648 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 10535
I20260924 19:01:17.784825 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276386554690-388-0/minicluster-data/ts-2/wal/instance
I20260924 19:01:17.785238 388 external_mini_cluster.cc:1428] Setting key 5ff009ae771f1d8de3e1175a80fc2a8b
I20260924 19:01:17.791026 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:17.785600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47753 (local address 127.0.97.62:39897)
0924 19:01:17.785834 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:17.785840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:17.787435 (+ 1595us) server_negotiation.cc:407] Connection header received
0924 19:01:17.789098 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:17.789102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:17.789150 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:17.789230 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:17.790478 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:01:17.790482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:01:17.790485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:01:17.790500 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:01:17.790530 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:01:17.790541 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:01:17.790635 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 19:01:17.790791 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 19:01:17.790839 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 19:01:17.791939 10664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:17.786196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39897 (local address 127.0.97.3:47753)
0924 19:01:17.787285 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:17.787328 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:17.788858 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:17.789428 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:17.789436 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:17.789593 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:17.790202 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:01:17.790204 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:17.790241 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:01:17.790334 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:01:17.790606 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:01:17.790611 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:01:17.790615 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:01:17.790836 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 19:01:17.791025 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":346,"threads_started":1}
I20260924 19:01:17.793229 10662 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39897
I20260924 19:01:17.793469 10662 heartbeater.cc:464] Registering TS with master...
I20260924 19:01:17.794049 10662 heartbeater.cc:511] Master 127.0.97.62:39897 requested a full tablet report, sending...
I20260924 19:01:17.795290 10216 ts_manager.cc:194] Registered new tserver with Master: a124b66f0f264af5adeaae72769d9f6d (127.0.97.3:41531)
I20260924 19:01:17.800195 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 19:01:17.805871 10254 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:01:17.804407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34760 (local address 127.0.97.62:39897)
0924 19:01:17.804559 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:17.804563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:17.805016 (+ 453us) server_negotiation.cc:407] Connection header received
0924 19:01:17.805160 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:17.805163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:17.805210 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:17.805285 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:17.805637 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34760: BlockingRecv error: recv got EOF from 127.0.0.1:34760 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 19:01:17.806196 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 10275
I20260924 19:01:17.809711 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:17.807210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54070 (local address 127.0.97.1:45381)
0924 19:01:17.807691 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:17.807697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:17.807720 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:01:17.807778 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:17.807784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:17.807962 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:17.808117 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:17.809036 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:01:17.809042 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:01:17.809053 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:01:17.809069 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:01:17.809104 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:01:17.809117 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:01:17.809262 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 19:01:17.809340 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 19:01:17.809430 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":203,"threads_started":1}
I20260924 19:01:17.810801 10397 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:01:17.890931 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 10275
I20260924 19:01:17.907475 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 10405
I20260924 19:01:17.911144 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:17.908370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57628 (local address 127.0.97.2:34619)
0924 19:01:17.908907 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:17.908914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:17.908942 (+ 28us) server_negotiation.cc:407] Connection header received
0924 19:01:17.909001 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:17.909008 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:17.909246 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:17.909379 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:17.910431 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:01:17.910437 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:01:17.910451 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:01:17.910463 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:01:17.910507 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:01:17.910524 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:01:17.910655 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 19:01:17.910802 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 19:01:17.910895 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":250,"threads_started":1}
I20260924 19:01:17.912724 10527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:01:17.994098 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 10405
I20260924 19:01:18.010828 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 10535
I20260924 19:01:18.014650 10675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:18.011874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46360 (local address 127.0.97.3:41531)
0924 19:01:18.012514 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:18.012520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:18.012550 (+ 30us) server_negotiation.cc:407] Connection header received
0924 19:01:18.012610 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:18.012616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:18.012830 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:18.012968 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:18.014020 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:01:18.014025 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:01:18.014037 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:01:18.014052 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:01:18.014088 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:01:18.014101 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:01:18.014250 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 19:01:18.014299 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 19:01:18.014392 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":347,"threads_started":1}
I20260924 19:01:18.015812 10657 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:01:18.096119 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 10535
I20260924 19:01:18.111984 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 10185
I20260924 19:01:18.113165 10246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:01:18.213871 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 10185
2026-09-24T19:01:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2291 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:01:18 dist-test-slave-9fn1 krb5kdc[10680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:01:18 dist-test-slave-9fn1 krb5kdc[10680](info): set up 2 sockets
Sep 24 19:01:18 dist-test-slave-9fn1 krb5kdc[10680](info): commencing operation
krb5kdc: starting...
W20260924 19:01:19.306262 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:01:19 dist-test-slave-9fn1 krb5kdc[10680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:01:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:01:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 19:01:19.521771 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41717
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41717
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:01:19.833220 10693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:01:19.833556 10693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:01:19.833716 10693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:01:19.840830 10693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:01:19.840927 10693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:01:19.841024 10693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:01:19.841120 10693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:01:19.841203 10693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:01:19.850587 10693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:01:19.851836 10693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:01:19.853152 10693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:01:19.864269 10701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:01:19.864192 10699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:01:19.864303 10698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:01:19.865103 10693 server_base.cc:1034] running on GCE node
I20260924 19:01:19.865898 10693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:01:19.867242 10693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:01:19.868436 10693 hybrid_clock.cc:648] HybridClock initialized: now 1790276479868401 us; error 30 us; skew 500 ppm
I20260924 19:01:19.868755 10693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:01:19 dist-test-slave-9fn1 krb5kdc[10680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276479, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:01:19.871722 10693 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:01:19.873057 10693 webserver.cc:460] Webserver started at http://127.0.97.62:40343/ using document root <none> and password file <none>
I20260924 19:01:19.873788 10693 fs_manager.cc:359] Metadata directory not provided
I20260924 19:01:19.873916 10693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:01:19.874233 10693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:01:19.877076 10693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "6db96d6813d64f448a6aa0f11579888f"
format_stamp: "Formatted at 2026-09-24 19:01:19 on dist-test-slave-9fn1"
server_key: "286ae266d503063b7149da4c9780a468"
server_key_iv: "8ed6bf8f29444dc285f1866fcd75f1d5"
server_key_version: "encryptionkey@0"
I20260924 19:01:19.877853 10693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "6db96d6813d64f448a6aa0f11579888f"
format_stamp: "Formatted at 2026-09-24 19:01:19 on dist-test-slave-9fn1"
server_key: "286ae266d503063b7149da4c9780a468"
server_key_iv: "8ed6bf8f29444dc285f1866fcd75f1d5"
server_key_version: "encryptionkey@0"
I20260924 19:01:19.882611 10693 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 19:01:19.886270 10708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:19.887511 10693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:01:19.887683 10693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "6db96d6813d64f448a6aa0f11579888f"
format_stamp: "Formatted at 2026-09-24 19:01:19 on dist-test-slave-9fn1"
server_key: "286ae266d503063b7149da4c9780a468"
server_key_iv: "8ed6bf8f29444dc285f1866fcd75f1d5"
server_key_version: "encryptionkey@0"
I20260924 19:01:19.887871 10693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:01:19.930838 10693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:01:19.934696 10693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:01:19.934967 10693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:01:19.952817 10693 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41717
I20260924 19:01:19.952836 10759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41717 every 8 connection(s)
I20260924 19:01:19.954222 10693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:01:19.957839 10760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:01:19.961984 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 10693
I20260924 19:01:19.962210 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:01:19.963011 388 external_mini_cluster.cc:1428] Setting key 0240c84cff292c115b63f066bdaa8e42
I20260924 19:01:19.969024 10760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6db96d6813d64f448a6aa0f11579888f: Bootstrap starting.
I20260924 19:01:19.972493 10760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6db96d6813d64f448a6aa0f11579888f: Neither blocks nor log segments found. Creating new log.
I20260924 19:01:19.973670 10760 log.cc:824] T 00000000000000000000000000000000 P 6db96d6813d64f448a6aa0f11579888f: Log is configured to *not* fsync() on all Append() calls
I20260924 19:01:19.976811 10760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6db96d6813d64f448a6aa0f11579888f: No bootstrap required, opened a new log
Sep 24 19:01:19 dist-test-slave-9fn1 krb5kdc[10680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:01:19.984406 10760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6db96d6813d64f448a6aa0f11579888f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6db96d6813d64f448a6aa0f11579888f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41717 } }
I20260924 19:01:19.984905 10760 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6db96d6813d64f448a6aa0f11579888f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:01:19.985023 10760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6db96d6813d64f448a6aa0f11579888f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6db96d6813d64f448a6aa0f11579888f, State: Initialized, Role: FOLLOWER
I20260924 19:01:19.985749 10760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6db96d6813d64f448a6aa0f11579888f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6db96d6813d64f448a6aa0f11579888f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41717 } }
I20260924 19:01:19.985996 10760 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6db96d6813d64f448a6aa0f11579888f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:01:19.986095 10763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:19.964937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33890 (local address 127.0.97.62:41717)
0924 19:01:19.965687 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:19.965701 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:19.965743 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:01:19.967610 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:19.967631 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:19.968003 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:19.968303 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:19.970052 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:19.971327 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:19.973124 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:19.973875 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:19.974016 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:19.978497 (+ 4481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:01:19.978504 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:01:19.978518 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:01:19.978553 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:01:19.981781 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:01:19.982504 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:01:19.982512 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:01:19.982519 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:01:19.982686 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:01:19.983089 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:01:19.983098 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:01:19.983104 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:01:19.983641 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:01:19.983923 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0924 19:01:19.984362 (+ 439us) server_negotiation.cc:299] Negotiation successful
0924 19:01:19.984816 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":346,"threads_started":1}
I20260924 19:01:19.986140 10760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6db96d6813d64f448a6aa0f11579888f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:01:19.986339 10760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6db96d6813d64f448a6aa0f11579888f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:01:19.988668 10760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6db96d6813d64f448a6aa0f11579888f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6db96d6813d64f448a6aa0f11579888f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41717 } }
I20260924 19:01:19.989250 10760 leader_election.cc:302] T 00000000000000000000000000000000 P 6db96d6813d64f448a6aa0f11579888f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6db96d6813d64f448a6aa0f11579888f; no voters:
I20260924 19:01:19.989869 10760 leader_election.cc:288] T 00000000000000000000000000000000 P 6db96d6813d64f448a6aa0f11579888f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:01:19.990015 10765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6db96d6813d64f448a6aa0f11579888f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:01:19.990779 10765 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6db96d6813d64f448a6aa0f11579888f [term 1 LEADER]: Becoming Leader. State: Replica: 6db96d6813d64f448a6aa0f11579888f, State: Running, Role: LEADER
I20260924 19:01:19.991325 10765 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6db96d6813d64f448a6aa0f11579888f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6db96d6813d64f448a6aa0f11579888f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41717 } }
I20260924 19:01:19.991964 10760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6db96d6813d64f448a6aa0f11579888f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:01:19.993991 10766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6db96d6813d64f448a6aa0f11579888f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6db96d6813d64f448a6aa0f11579888f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6db96d6813d64f448a6aa0f11579888f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41717 } } }
I20260924 19:01:19.994341 10766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6db96d6813d64f448a6aa0f11579888f [sys.catalog]: This master's current role is: LEADER
I20260924 19:01:19.994817 10767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6db96d6813d64f448a6aa0f11579888f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6db96d6813d64f448a6aa0f11579888f. Latest consensus state: current_term: 1 leader_uuid: "6db96d6813d64f448a6aa0f11579888f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6db96d6813d64f448a6aa0f11579888f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41717 } } }
I20260924 19:01:19.995136 10767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6db96d6813d64f448a6aa0f11579888f [sys.catalog]: This master's current role is: LEADER
I20260924 19:01:19.999718 10771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:01:20.005020 10771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:01:20.013321 10771 catalog_manager.cc:1347] Generated new cluster ID: 255e3c72f82741c1a87feefa81c90716
I20260924 19:01:20.013441 10771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:01:20.029697 10771 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:01:20.031337 10771 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:01:20.036761 10771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6db96d6813d64f448a6aa0f11579888f: Generated new TSK 0
I20260924 19:01:20.037617 10771 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 19:01:20.092730 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41717
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:01:20.417393 10787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:01:20.417701 10787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:01:20.417858 10787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:01:20.425586 10787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:01:20.425745 10787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:01:20.425940 10787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:01:20.435407 10787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:01:20.436785 10787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:01:20.438159 10787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:01:20.451545 10792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:01:20.451614 10793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:01:20.451786 10795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:01:20.452630 10787 server_base.cc:1034] running on GCE node
I20260924 19:01:20.453267 10787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:01:20.454286 10787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:01:20.455529 10787 hybrid_clock.cc:648] HybridClock initialized: now 1790276480455481 us; error 39 us; skew 500 ppm
I20260924 19:01:20.455848 10787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:01:20 dist-test-slave-9fn1 krb5kdc[10680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:01:20.459100 10787 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 19:01:20.460620 10787 webserver.cc:460] Webserver started at http://127.0.97.1:39555/ using document root <none> and password file <none>
I20260924 19:01:20.461278 10787 fs_manager.cc:359] Metadata directory not provided
I20260924 19:01:20.461383 10787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:01:20.461706 10787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:01:20.464524 10787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-0/data/instance:
uuid: "9bce5a795a3d4e8aa276133ca66fcc58"
format_stamp: "Formatted at 2026-09-24 19:01:20 on dist-test-slave-9fn1"
server_key: "1c400c2ceea00ad7e067a80f08991d1e"
server_key_iv: "8ef10fe0047db60999023b1aa5058e83"
server_key_version: "encryptionkey@0"
I20260924 19:01:20.465238 10787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9bce5a795a3d4e8aa276133ca66fcc58"
format_stamp: "Formatted at 2026-09-24 19:01:20 on dist-test-slave-9fn1"
server_key: "1c400c2ceea00ad7e067a80f08991d1e"
server_key_iv: "8ef10fe0047db60999023b1aa5058e83"
server_key_version: "encryptionkey@0"
I20260924 19:01:20.470284 10787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260924 19:01:20.473619 10802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:20.474997 10787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 19:01:20.475167 10787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "9bce5a795a3d4e8aa276133ca66fcc58"
format_stamp: "Formatted at 2026-09-24 19:01:20 on dist-test-slave-9fn1"
server_key: "1c400c2ceea00ad7e067a80f08991d1e"
server_key_iv: "8ef10fe0047db60999023b1aa5058e83"
server_key_version: "encryptionkey@0"
I20260924 19:01:20.475392 10787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:01:20.495137 10787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:01:20.499648 10787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:01:20.499951 10787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:01:20.500857 10787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:01:20.502199 10787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:01:20.502301 10787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:20.502465 10787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:01:20.502539 10787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:20.534443 10787 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41233
I20260924 19:01:20.534478 10914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41233 every 8 connection(s)
I20260924 19:01:20.535697 10787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:01:20.544525 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 10787
I20260924 19:01:20.544761 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-0/wal/instance
I20260924 19:01:20.545360 388 external_mini_cluster.cc:1428] Setting key 366a2606c48a20fdca4d822522b33734
Sep 24 19:01:20 dist-test-slave-9fn1 krb5kdc[10680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:01:20.560370 10917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:20.538253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41955 (local address 127.0.97.62:41717)
0924 19:01:20.538667 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:20.538672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:20.539720 (+ 1048us) server_negotiation.cc:407] Connection header received
0924 19:01:20.541468 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:20.541474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:20.541613 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:20.541741 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:20.543170 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:20.543906 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:20.545565 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:20.545946 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:20.546074 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:20.554575 (+ 8501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:01:20.554580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:01:20.554583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:01:20.554612 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:01:20.557169 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:01:20.558141 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:01:20.558154 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:01:20.558156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:01:20.558262 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:01:20.558742 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:01:20.558745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:01:20.558746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:01:20.558989 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:01:20.559086 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 19:01:20.559977 (+ 891us) server_negotiation.cc:299] Negotiation successful
0924 19:01:20.560126 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":228,"threads_started":1}
I20260924 19:01:20.562071 10918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:20.538770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41717 (local address 127.0.97.1:41955)
0924 19:01:20.539515 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:20.539553 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:20.541205 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:20.541965 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:20.541973 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:20.542478 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:20.543007 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:20.543023 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:20.544075 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:20.544079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:20.545382 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:20.545394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:20.546170 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:20.546177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:20.546436 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:20.551760 (+ 5324us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:20.551787 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:20.554328 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:01:20.557330 (+ 3002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:01:20.557341 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:01:20.557361 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:01:20.558010 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:01:20.558375 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:01:20.558382 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:01:20.558388 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:01:20.558621 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:01:20.559110 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:01:20.559120 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:01:20.559570 (+ 450us) client_negotiation.cc:769] Sending connection context
0924 19:01:20.559891 (+ 321us) client_negotiation.cc:240] Negotiation successful
0924 19:01:20.560451 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":288,"threads_started":1}
I20260924 19:01:20.563757 10915 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41717
I20260924 19:01:20.564112 10915 heartbeater.cc:464] Registering TS with master...
I20260924 19:01:20.564844 10915 heartbeater.cc:511] Master 127.0.97.62:41717 requested a full tablet report, sending...
I20260924 19:01:20.567216 10725 ts_manager.cc:194] Registered new tserver with Master: 9bce5a795a3d4e8aa276133ca66fcc58 (127.0.97.1:41233)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 19:01:20.569185 10725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41955
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 19:01:20.596247 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41717
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:01:20.914472 10921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:01:20.914739 10921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:01:20.914830 10921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:01:20.922039 10921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:01:20.922147 10921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:01:20.922256 10921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:01:20.931663 10921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:01:20.932837 10921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:01:20.934073 10921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:01:20.946903 10927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:01:20.946826 10926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:01:20.947445 10929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:01:20.947685 10921 server_base.cc:1034] running on GCE node
I20260924 19:01:20.948279 10921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:01:20.949163 10921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:01:20.950350 10921 hybrid_clock.cc:648] HybridClock initialized: now 1790276480950314 us; error 32 us; skew 500 ppm
I20260924 19:01:20.950616 10921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:01:20 dist-test-slave-9fn1 krb5kdc[10680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:01:20.953720 10921 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 19:01:20.955241 10921 webserver.cc:460] Webserver started at http://127.0.97.2:38123/ using document root <none> and password file <none>
I20260924 19:01:20.955931 10921 fs_manager.cc:359] Metadata directory not provided
I20260924 19:01:20.956071 10921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:01:20.956427 10921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:01:20.959154 10921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-1/data/instance:
uuid: "2e51951ff485489b8f1cab74aad4db5e"
format_stamp: "Formatted at 2026-09-24 19:01:20 on dist-test-slave-9fn1"
server_key: "2903a6cabaf89d103368aaa4328a889b"
server_key_iv: "7fcce3ff848487fac81e80fe7102d67d"
server_key_version: "encryptionkey@0"
I20260924 19:01:20.959918 10921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2e51951ff485489b8f1cab74aad4db5e"
format_stamp: "Formatted at 2026-09-24 19:01:20 on dist-test-slave-9fn1"
server_key: "2903a6cabaf89d103368aaa4328a889b"
server_key_iv: "7fcce3ff848487fac81e80fe7102d67d"
server_key_version: "encryptionkey@0"
I20260924 19:01:20.964828 10921 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 19:01:20.968272 10936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:20.969578 10921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:01:20.969739 10921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "2e51951ff485489b8f1cab74aad4db5e"
format_stamp: "Formatted at 2026-09-24 19:01:20 on dist-test-slave-9fn1"
server_key: "2903a6cabaf89d103368aaa4328a889b"
server_key_iv: "7fcce3ff848487fac81e80fe7102d67d"
server_key_version: "encryptionkey@0"
I20260924 19:01:20.969924 10921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:01:20.985376 10921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:01:20.989099 10921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:01:20.989384 10921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:01:20.990329 10921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:01:20.991693 10921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:01:20.991786 10921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:20.991936 10921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:01:20.992007 10921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:21.024816 10921 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45343
I20260924 19:01:21.024861 11048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45343 every 8 connection(s)
I20260924 19:01:21.026093 10921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:01:21.027571 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 10921
I20260924 19:01:21.027766 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-1/wal/instance
I20260924 19:01:21.028187 388 external_mini_cluster.cc:1428] Setting key 03298ce090d2b73a1942808e18a0a2b1
Sep 24 19:01:21 dist-test-slave-9fn1 krb5kdc[10680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:01:21.045640 10917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:21.028615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37855 (local address 127.0.97.62:41717)
0924 19:01:21.028842 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:21.028849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:21.030247 (+ 1398us) server_negotiation.cc:407] Connection header received
0924 19:01:21.032021 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:21.032026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:21.032110 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:21.032228 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:21.033802 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.034511 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.036075 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.036483 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.036616 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:21.040114 (+ 3498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:01:21.040119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:01:21.040123 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:01:21.040154 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:01:21.042641 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:01:21.043384 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:01:21.043397 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:01:21.043400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:01:21.043518 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:01:21.043948 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:01:21.043952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:01:21.043956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:01:21.044210 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:01:21.044359 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 19:01:21.045032 (+ 673us) server_negotiation.cc:299] Negotiation successful
0924 19:01:21.045252 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 19:01:21.046665 11051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:21.029243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41717 (local address 127.0.97.2:37855)
0924 19:01:21.030088 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:21.030148 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:21.031750 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:21.032469 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:21.032478 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:21.033005 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:21.033630 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:21.033649 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.034666 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.034671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:21.035919 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:21.035931 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.036706 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.036712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:21.036953 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:21.037725 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:21.037748 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:21.039889 (+ 2141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:01:21.042787 (+ 2898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:01:21.042798 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:01:21.042815 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:01:21.043250 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:01:21.043627 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:01:21.043634 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:01:21.043637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:01:21.043826 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:01:21.044344 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:01:21.044352 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:01:21.044690 (+ 338us) client_negotiation.cc:769] Sending connection context
0924 19:01:21.044951 (+ 261us) client_negotiation.cc:240] Negotiation successful
0924 19:01:21.045491 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":314,"threads_started":1}
I20260924 19:01:21.048103 11049 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41717
I20260924 19:01:21.048403 11049 heartbeater.cc:464] Registering TS with master...
I20260924 19:01:21.049110 11049 heartbeater.cc:511] Master 127.0.97.62:41717 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 19:01:21.050616 10725 ts_manager.cc:194] Registered new tserver with Master: 2e51951ff485489b8f1cab74aad4db5e (127.0.97.2:45343)
I20260924 19:01:21.051462 10725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 19:01:21.078049 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41717
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:01:21.396054 11054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:01:21.396308 11054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:01:21.396399 11054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:01:21.403486 11054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:01:21.403582 11054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:01:21.403694 11054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:01:21.413041 11054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41717
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:01:21.414198 11054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:01:21.415465 11054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:01:21.428229 11062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:01:21.428352 11059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:01:21.428331 11060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:01:21.429255 11054 server_base.cc:1034] running on GCE node
I20260924 19:01:21.429908 11054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:01:21.430847 11054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:01:21.432071 11054 hybrid_clock.cc:648] HybridClock initialized: now 1790276481432019 us; error 47 us; skew 500 ppm
I20260924 19:01:21.432390 11054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:01:21 dist-test-slave-9fn1 krb5kdc[10680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:01:21.435606 11054 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 19:01:21.437084 11054 webserver.cc:460] Webserver started at http://127.0.97.3:41605/ using document root <none> and password file <none>
I20260924 19:01:21.437876 11054 fs_manager.cc:359] Metadata directory not provided
I20260924 19:01:21.438023 11054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:01:21.438339 11054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:01:21.441253 11054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-2/data/instance:
uuid: "845e8192bd174d4e96d4f0a5fdf397f0"
format_stamp: "Formatted at 2026-09-24 19:01:21 on dist-test-slave-9fn1"
server_key: "4a4d9c4126452a9d3ccfa33fafd1c5ca"
server_key_iv: "b3644232dfc748caed1f3499d78f8158"
server_key_version: "encryptionkey@0"
I20260924 19:01:21.442101 11054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-2/wal/instance:
uuid: "845e8192bd174d4e96d4f0a5fdf397f0"
format_stamp: "Formatted at 2026-09-24 19:01:21 on dist-test-slave-9fn1"
server_key: "4a4d9c4126452a9d3ccfa33fafd1c5ca"
server_key_iv: "b3644232dfc748caed1f3499d78f8158"
server_key_version: "encryptionkey@0"
I20260924 19:01:21.447281 11054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:01:21.450771 11069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:21.452093 11054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:01:21.452263 11054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "845e8192bd174d4e96d4f0a5fdf397f0"
format_stamp: "Formatted at 2026-09-24 19:01:21 on dist-test-slave-9fn1"
server_key: "4a4d9c4126452a9d3ccfa33fafd1c5ca"
server_key_iv: "b3644232dfc748caed1f3499d78f8158"
server_key_version: "encryptionkey@0"
I20260924 19:01:21.452476 11054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:01:21.483516 11054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:01:21.487412 11054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:01:21.487702 11054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:01:21.488658 11054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:01:21.490053 11054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:01:21.490154 11054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:21.490304 11054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:01:21.490376 11054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:01:21.523664 11054 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46667
I20260924 19:01:21.523705 11181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46667 every 8 connection(s)
I20260924 19:01:21.524947 11054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:01:21.528121 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 11054
I20260924 19:01:21.528342 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-2/wal/instance
I20260924 19:01:21.528761 388 external_mini_cluster.cc:1428] Setting key 6067b66b0c6f00b716e5891585fbefe0
Sep 24 19:01:21 dist-test-slave-9fn1 krb5kdc[10680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:01:21.544111 10917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:21.527387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51609 (local address 127.0.97.62:41717)
0924 19:01:21.527556 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:21.527561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:21.528918 (+ 1357us) server_negotiation.cc:407] Connection header received
0924 19:01:21.530944 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:21.530949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:21.531025 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:21.531129 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:21.532587 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.533326 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.534859 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.535265 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.535400 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:21.538726 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:01:21.538733 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:01:21.538737 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:01:21.538793 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:01:21.541054 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:01:21.541781 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:01:21.541786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:01:21.541790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:01:21.541917 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:01:21.542314 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:01:21.542318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:01:21.542320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:01:21.542692 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:01:21.542841 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 19:01:21.543655 (+ 814us) server_negotiation.cc:299] Negotiation successful
0924 19:01:21.543872 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 19:01:21.544961 11184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:21.527897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41717 (local address 127.0.97.3:51609)
0924 19:01:21.528667 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:21.528723 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:21.530663 (+ 1940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:21.531306 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:21.531313 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:21.531910 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:01:21.532433 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:21.532450 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.533450 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.533454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:21.534704 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:21.534715 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.535485 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.535494 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:21.535716 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:21.536370 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:01:21.536394 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:01:21.538542 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:01:21.541192 (+ 2650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:01:21.541199 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:01:21.541212 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:01:21.541665 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:01:21.542021 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:01:21.542034 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:01:21.542038 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:01:21.542201 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:01:21.542835 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:01:21.542842 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:01:21.543234 (+ 392us) client_negotiation.cc:769] Sending connection context
0924 19:01:21.543489 (+ 255us) client_negotiation.cc:240] Negotiation successful
0924 19:01:21.543968 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":345,"threads_started":1}
I20260924 19:01:21.546116 11182 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41717
I20260924 19:01:21.546392 11182 heartbeater.cc:464] Registering TS with master...
I20260924 19:01:21.547044 11182 heartbeater.cc:511] Master 127.0.97.62:41717 requested a full tablet report, sending...
I20260924 19:01:21.548316 10725 ts_manager.cc:194] Registered new tserver with Master: 845e8192bd174d4e96d4f0a5fdf397f0 (127.0.97.3:46667)
I20260924 19:01:21.549033 10725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51609
I20260924 19:01:21.555735 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:01:21.571610 10917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:21.560021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33904 (local address 127.0.97.62:41717)
0924 19:01:21.560207 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:21.560216 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:21.560676 (+ 460us) server_negotiation.cc:407] Connection header received
0924 19:01:21.560803 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:21.560806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:21.560863 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:21.560939 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:21.562281 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.562952 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.564173 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.564534 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.564621 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:21.566770 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:01:21.566774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:01:21.566778 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:01:21.566807 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:01:21.569247 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:01:21.569934 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:01:21.569938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:01:21.569940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:01:21.570048 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:01:21.570388 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:01:21.570391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:01:21.570393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:01:21.570667 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:01:21.570831 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 19:01:21.571123 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 19:01:21.571279 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 19:01:21.572322 10915 heartbeater.cc:503] Master 127.0.97.62:41717 was elected leader, sending a full tablet report...
I20260924 19:01:21.575564 10725 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:01:21.582477 10725 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:01:21.607244 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:21.593437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46667 (local address 127.0.0.1:40218)
0924 19:01:21.595037 (+ 1600us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:21.595097 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:21.595272 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:21.596372 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:21.596383 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:21.596421 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:01:21.596909 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:21.596924 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.599595 (+ 2671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.599604 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:21.602832 (+ 3228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:21.602852 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.605872 (+ 3020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.605882 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:21.606219 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:21.606299 (+ 80us) client_negotiation.cc:769] Sending connection context
0924 19:01:21.606438 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 19:01:21.606698 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1371,"thread_start_us":13035,"threads_started":1}
I20260924 19:01:21.607440 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:21.593824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40218 (local address 127.0.97.3:46667)
0924 19:01:21.594567 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:21.594573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:21.595454 (+ 881us) server_negotiation.cc:407] Connection header received
0924 19:01:21.595569 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:21.595576 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:21.595818 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:21.596004 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:01:21.597174 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.599410 (+ 2236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.603059 (+ 3649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.605013 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.605179 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:21.606797 (+ 1618us) server_negotiation.cc:1035] Waiting for connection context
0924 19:01:21.606878 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 19:01:21.607056 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":756,"threads_started":1}
I20260924 19:01:21.611536 11117 tablet_service.cc:1467] Processing CreateTablet for tablet 0033afea97f447dc99cd5154dc3cdb82 (DEFAULT_TABLE table=test-table [id=a7e7af552fc5469c9f2f0fa0dc2b9322]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:01:21.613018 11117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0033afea97f447dc99cd5154dc3cdb82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:01:21.615737 11199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:21.595812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48352 (local address 127.0.97.2:45343)
0924 19:01:21.608912 (+ 13100us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:21.608920 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:21.608949 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:01:21.609016 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:21.609021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:21.609176 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:21.609366 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:01:21.609918 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.611393 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.613299 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.614666 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.614809 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:21.614999 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 19:01:21.615216 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 19:01:21.615439 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12923,"thread_start_us":12828,"threads_started":1}
I20260924 19:01:21.615773 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:21.595629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45343 (local address 127.0.0.1:48352)
0924 19:01:21.606489 (+ 10860us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:21.606519 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:21.606623 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:21.609354 (+ 2731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:21.609358 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:21.609381 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:01:21.609723 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:21.609731 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.611530 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.611536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:21.613094 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:21.613108 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.614860 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.614868 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:21.615021 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:21.615061 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 19:01:21.615161 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 19:01:21.615318 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10724,"mutex_wait_us":49,"thread_start_us":10270,"threads_started":1}
I20260924 19:01:21.616379 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:21.594737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35590 (local address 127.0.97.1:41233)
0924 19:01:21.595362 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:21.595373 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:21.606334 (+ 10961us) server_negotiation.cc:407] Connection header received
0924 19:01:21.607579 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:21.607587 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:21.607870 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:21.608026 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:01:21.608904 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.610658 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.613065 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.614774 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.614876 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:21.615047 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 19:01:21.615865 (+ 818us) server_negotiation.cc:299] Negotiation successful
0924 19:01:21.616053 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":297,"threads_started":1}
I20260924 19:01:21.616511 11198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:21.594460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41233 (local address 127.0.0.1:35590)
0924 19:01:21.606208 (+ 11748us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:21.606238 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:21.606381 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:21.608321 (+ 1940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:21.608325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:21.608349 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:01:21.608681 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:21.608691 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.610840 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.610845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:21.612886 (+ 2041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:21.612899 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.615212 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.615218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:21.615444 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:21.615478 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 19:01:21.615564 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 19:01:21.615688 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11589,"mutex_wait_us":28,"thread_start_us":319,"threads_started":1}
I20260924 19:01:21.619684 10850 tablet_service.cc:1467] Processing CreateTablet for tablet 0033afea97f447dc99cd5154dc3cdb82 (DEFAULT_TABLE table=test-table [id=a7e7af552fc5469c9f2f0fa0dc2b9322]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:01:21.619910 10984 tablet_service.cc:1467] Processing CreateTablet for tablet 0033afea97f447dc99cd5154dc3cdb82 (DEFAULT_TABLE table=test-table [id=a7e7af552fc5469c9f2f0fa0dc2b9322]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:01:21.620913 10850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0033afea97f447dc99cd5154dc3cdb82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:01:21.621156 10984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0033afea97f447dc99cd5154dc3cdb82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:01:21.627701 11200 tablet_bootstrap.cc:492] T 0033afea97f447dc99cd5154dc3cdb82 P 845e8192bd174d4e96d4f0a5fdf397f0: Bootstrap starting.
I20260924 19:01:21.631803 11200 tablet_bootstrap.cc:654] T 0033afea97f447dc99cd5154dc3cdb82 P 845e8192bd174d4e96d4f0a5fdf397f0: Neither blocks nor log segments found. Creating new log.
I20260924 19:01:21.633193 11200 log.cc:824] T 0033afea97f447dc99cd5154dc3cdb82 P 845e8192bd174d4e96d4f0a5fdf397f0: Log is configured to *not* fsync() on all Append() calls
I20260924 19:01:21.633551 11201 tablet_bootstrap.cc:492] T 0033afea97f447dc99cd5154dc3cdb82 P 2e51951ff485489b8f1cab74aad4db5e: Bootstrap starting.
I20260924 19:01:21.635106 11202 tablet_bootstrap.cc:492] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58: Bootstrap starting.
I20260924 19:01:21.636909 11200 tablet_bootstrap.cc:492] T 0033afea97f447dc99cd5154dc3cdb82 P 845e8192bd174d4e96d4f0a5fdf397f0: No bootstrap required, opened a new log
I20260924 19:01:21.637195 11200 ts_tablet_manager.cc:1397] T 0033afea97f447dc99cd5154dc3cdb82 P 845e8192bd174d4e96d4f0a5fdf397f0: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260924 19:01:21.637205 11201 tablet_bootstrap.cc:654] T 0033afea97f447dc99cd5154dc3cdb82 P 2e51951ff485489b8f1cab74aad4db5e: Neither blocks nor log segments found. Creating new log.
I20260924 19:01:21.638427 11202 tablet_bootstrap.cc:654] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58: Neither blocks nor log segments found. Creating new log.
I20260924 19:01:21.638427 11201 log.cc:824] T 0033afea97f447dc99cd5154dc3cdb82 P 2e51951ff485489b8f1cab74aad4db5e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:01:21.639578 11202 log.cc:824] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58: Log is configured to *not* fsync() on all Append() calls
I20260924 19:01:21.641549 11201 tablet_bootstrap.cc:492] T 0033afea97f447dc99cd5154dc3cdb82 P 2e51951ff485489b8f1cab74aad4db5e: No bootstrap required, opened a new log
I20260924 19:01:21.641808 11201 ts_tablet_manager.cc:1397] T 0033afea97f447dc99cd5154dc3cdb82 P 2e51951ff485489b8f1cab74aad4db5e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260924 19:01:21.642196 11202 tablet_bootstrap.cc:492] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58: No bootstrap required, opened a new log
I20260924 19:01:21.642438 11202 ts_tablet_manager.cc:1397] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260924 19:01:21.647585 11200 raft_consensus.cc:348] T 0033afea97f447dc99cd5154dc3cdb82 P 845e8192bd174d4e96d4f0a5fdf397f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "845e8192bd174d4e96d4f0a5fdf397f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46667 } } peers { permanent_uuid: "9bce5a795a3d4e8aa276133ca66fcc58" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41233 } } peers { permanent_uuid: "2e51951ff485489b8f1cab74aad4db5e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45343 } }
I20260924 19:01:21.648030 11200 raft_consensus.cc:374] T 0033afea97f447dc99cd5154dc3cdb82 P 845e8192bd174d4e96d4f0a5fdf397f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:01:21.648144 11200 raft_consensus.cc:729] T 0033afea97f447dc99cd5154dc3cdb82 P 845e8192bd174d4e96d4f0a5fdf397f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 845e8192bd174d4e96d4f0a5fdf397f0, State: Initialized, Role: FOLLOWER
I20260924 19:01:21.648777 11200 consensus_queue.cc:260] T 0033afea97f447dc99cd5154dc3cdb82 P 845e8192bd174d4e96d4f0a5fdf397f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "845e8192bd174d4e96d4f0a5fdf397f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46667 } } peers { permanent_uuid: "9bce5a795a3d4e8aa276133ca66fcc58" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41233 } } peers { permanent_uuid: "2e51951ff485489b8f1cab74aad4db5e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45343 } }
I20260924 19:01:21.649982 11182 heartbeater.cc:503] Master 127.0.97.62:41717 was elected leader, sending a full tablet report...
I20260924 19:01:21.650449 11202 raft_consensus.cc:348] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "845e8192bd174d4e96d4f0a5fdf397f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46667 } } peers { permanent_uuid: "9bce5a795a3d4e8aa276133ca66fcc58" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41233 } } peers { permanent_uuid: "2e51951ff485489b8f1cab74aad4db5e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45343 } }
I20260924 19:01:21.650648 11200 ts_tablet_manager.cc:1428] T 0033afea97f447dc99cd5154dc3cdb82 P 845e8192bd174d4e96d4f0a5fdf397f0: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260924 19:01:21.650771 11202 raft_consensus.cc:374] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:01:21.650880 11202 raft_consensus.cc:729] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bce5a795a3d4e8aa276133ca66fcc58, State: Initialized, Role: FOLLOWER
I20260924 19:01:21.651460 11202 consensus_queue.cc:260] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "845e8192bd174d4e96d4f0a5fdf397f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46667 } } peers { permanent_uuid: "9bce5a795a3d4e8aa276133ca66fcc58" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41233 } } peers { permanent_uuid: "2e51951ff485489b8f1cab74aad4db5e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45343 } }
I20260924 19:01:21.653385 11201 raft_consensus.cc:348] T 0033afea97f447dc99cd5154dc3cdb82 P 2e51951ff485489b8f1cab74aad4db5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "845e8192bd174d4e96d4f0a5fdf397f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46667 } } peers { permanent_uuid: "9bce5a795a3d4e8aa276133ca66fcc58" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41233 } } peers { permanent_uuid: "2e51951ff485489b8f1cab74aad4db5e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45343 } }
I20260924 19:01:21.653820 11201 raft_consensus.cc:374] T 0033afea97f447dc99cd5154dc3cdb82 P 2e51951ff485489b8f1cab74aad4db5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:01:21.653928 11201 raft_consensus.cc:729] T 0033afea97f447dc99cd5154dc3cdb82 P 2e51951ff485489b8f1cab74aad4db5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e51951ff485489b8f1cab74aad4db5e, State: Initialized, Role: FOLLOWER
I20260924 19:01:21.654137 11202 ts_tablet_manager.cc:1428] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s
I20260924 19:01:21.654515 11201 consensus_queue.cc:260] T 0033afea97f447dc99cd5154dc3cdb82 P 2e51951ff485489b8f1cab74aad4db5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "845e8192bd174d4e96d4f0a5fdf397f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46667 } } peers { permanent_uuid: "9bce5a795a3d4e8aa276133ca66fcc58" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41233 } } peers { permanent_uuid: "2e51951ff485489b8f1cab74aad4db5e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45343 } }
I20260924 19:01:21.655598 11049 heartbeater.cc:503] Master 127.0.97.62:41717 was elected leader, sending a full tablet report...
I20260924 19:01:21.656167 11201 ts_tablet_manager.cc:1428] T 0033afea97f447dc99cd5154dc3cdb82 P 2e51951ff485489b8f1cab74aad4db5e: Time spent starting tablet: real 0.014s user 0.011s sys 0.001s
I20260924 19:01:21.707908 11207 raft_consensus.cc:482] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:01:21.708168 11207 raft_consensus.cc:504] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "845e8192bd174d4e96d4f0a5fdf397f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46667 } } peers { permanent_uuid: "9bce5a795a3d4e8aa276133ca66fcc58" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41233 } } peers { permanent_uuid: "2e51951ff485489b8f1cab74aad4db5e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45343 } }
I20260924 19:01:21.709789 11207 leader_election.cc:288] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 845e8192bd174d4e96d4f0a5fdf397f0 (127.0.97.3:46667), 2e51951ff485489b8f1cab74aad4db5e (127.0.97.2:45343)
I20260924 19:01:21.716404 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:21.710133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44181 (local address 127.0.97.3:46667)
0924 19:01:21.710359 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:21.710368 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:21.711188 (+ 820us) server_negotiation.cc:407] Connection header received
0924 19:01:21.711304 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:21.711308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:21.711379 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:21.711460 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:01:21.712032 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.712778 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.714690 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.715450 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.715595 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:21.715712 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 19:01:21.716034 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 19:01:21.716153 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 19:01:21.716549 11199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:21.710206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43575 (local address 127.0.97.2:45343)
0924 19:01:21.710582 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:21.710590 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:21.710855 (+ 265us) server_negotiation.cc:407] Connection header received
0924 19:01:21.711094 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:21.711099 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:21.711192 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:21.711272 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:01:21.712153 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.712912 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.714693 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.715436 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.715596 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:21.715694 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 19:01:21.716164 (+ 470us) server_negotiation.cc:299] Negotiation successful
0924 19:01:21.716297 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260924 19:01:21.716708 11209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:21.710032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45343 (local address 127.0.97.1:43575)
0924 19:01:21.710746 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:21.710762 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:21.710937 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:21.711584 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:21.711587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:21.711617 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:01:21.711991 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:21.711998 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.713058 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.713063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:21.714546 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:21.714557 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.715674 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.715681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:21.715815 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:21.715837 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 19:01:21.716139 (+ 302us) client_negotiation.cc:240] Negotiation successful
0924 19:01:21.716278 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":2,"thread_start_us":422,"threads_started":1}
I20260924 19:01:21.716959 11210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:21.710112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46667 (local address 127.0.97.1:44181)
0924 19:01:21.711087 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0924 19:01:21.711100 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:01:21.711203 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:01:21.711485 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:01:21.711488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:01:21.711506 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:01:21.711868 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:21.711875 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.712904 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.712912 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:21.714539 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:01:21.714550 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.715718 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.715724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:01:21.715861 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:21.715900 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 19:01:21.716006 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 19:01:21.716123 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"mutex_wait_us":82,"thread_start_us":654,"threads_started":1}
I20260924 19:01:21.717640 11004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0033afea97f447dc99cd5154dc3cdb82" candidate_uuid: "9bce5a795a3d4e8aa276133ca66fcc58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e51951ff485489b8f1cab74aad4db5e" is_pre_election: true
I20260924 19:01:21.718129 11004 raft_consensus.cc:2454] T 0033afea97f447dc99cd5154dc3cdb82 P 2e51951ff485489b8f1cab74aad4db5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bce5a795a3d4e8aa276133ca66fcc58 in term 0.
I20260924 19:01:21.718242 11137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0033afea97f447dc99cd5154dc3cdb82" candidate_uuid: "9bce5a795a3d4e8aa276133ca66fcc58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "845e8192bd174d4e96d4f0a5fdf397f0" is_pre_election: true
I20260924 19:01:21.718725 11137 raft_consensus.cc:2454] T 0033afea97f447dc99cd5154dc3cdb82 P 845e8192bd174d4e96d4f0a5fdf397f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bce5a795a3d4e8aa276133ca66fcc58 in term 0.
I20260924 19:01:21.718745 10805 leader_election.cc:302] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e51951ff485489b8f1cab74aad4db5e, 9bce5a795a3d4e8aa276133ca66fcc58; no voters:
I20260924 19:01:21.719100 11207 raft_consensus.cc:2781] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:01:21.719250 11207 raft_consensus.cc:482] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:01:21.719390 11207 raft_consensus.cc:3037] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:01:21.721549 11207 raft_consensus.cc:504] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "845e8192bd174d4e96d4f0a5fdf397f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46667 } } peers { permanent_uuid: "9bce5a795a3d4e8aa276133ca66fcc58" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41233 } } peers { permanent_uuid: "2e51951ff485489b8f1cab74aad4db5e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45343 } }
I20260924 19:01:21.722196 11207 leader_election.cc:288] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58 [CANDIDATE]: Term 1 election: Requested vote from peers 845e8192bd174d4e96d4f0a5fdf397f0 (127.0.97.3:46667), 2e51951ff485489b8f1cab74aad4db5e (127.0.97.2:45343)
I20260924 19:01:21.722517 11137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0033afea97f447dc99cd5154dc3cdb82" candidate_uuid: "9bce5a795a3d4e8aa276133ca66fcc58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "845e8192bd174d4e96d4f0a5fdf397f0"
I20260924 19:01:21.722590 11004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0033afea97f447dc99cd5154dc3cdb82" candidate_uuid: "9bce5a795a3d4e8aa276133ca66fcc58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e51951ff485489b8f1cab74aad4db5e"
I20260924 19:01:21.722699 11137 raft_consensus.cc:3037] T 0033afea97f447dc99cd5154dc3cdb82 P 845e8192bd174d4e96d4f0a5fdf397f0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:01:21.722776 11004 raft_consensus.cc:3037] T 0033afea97f447dc99cd5154dc3cdb82 P 2e51951ff485489b8f1cab74aad4db5e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:01:21.724799 11137 raft_consensus.cc:2454] T 0033afea97f447dc99cd5154dc3cdb82 P 845e8192bd174d4e96d4f0a5fdf397f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bce5a795a3d4e8aa276133ca66fcc58 in term 1.
I20260924 19:01:21.724870 11004 raft_consensus.cc:2454] T 0033afea97f447dc99cd5154dc3cdb82 P 2e51951ff485489b8f1cab74aad4db5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bce5a795a3d4e8aa276133ca66fcc58 in term 1.
I20260924 19:01:21.725266 10805 leader_election.cc:302] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e51951ff485489b8f1cab74aad4db5e, 9bce5a795a3d4e8aa276133ca66fcc58; no voters:
I20260924 19:01:21.725558 11207 raft_consensus.cc:2781] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:01:21.725978 11207 raft_consensus.cc:686] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58 [term 1 LEADER]: Becoming Leader. State: Replica: 9bce5a795a3d4e8aa276133ca66fcc58, State: Running, Role: LEADER
I20260924 19:01:21.726460 11207 consensus_queue.cc:237] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "845e8192bd174d4e96d4f0a5fdf397f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46667 } } peers { permanent_uuid: "9bce5a795a3d4e8aa276133ca66fcc58" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41233 } } peers { permanent_uuid: "2e51951ff485489b8f1cab74aad4db5e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45343 } }
I20260924 19:01:21.730736 10723 catalog_manager.cc:5495] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bce5a795a3d4e8aa276133ca66fcc58 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "9bce5a795a3d4e8aa276133ca66fcc58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "845e8192bd174d4e96d4f0a5fdf397f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bce5a795a3d4e8aa276133ca66fcc58" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e51951ff485489b8f1cab74aad4db5e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45343 } health_report { overall_health: UNKNOWN } } }
I20260924 19:01:21.771127 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:21.766016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35596 (local address 127.0.97.1:41233)
0924 19:01:21.766236 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:21.766242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:21.766257 (+ 15us) server_negotiation.cc:407] Connection header received
0924 19:01:21.766339 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:21.766343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:21.766427 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:21.766584 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:01:21.766958 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.767663 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.768990 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.769391 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.769540 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:21.770028 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:01:21.770528 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:01:21.770687 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 19:01:21.770800 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 19:01:21.770873 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 19:01:21.776717 11183 tablet.cc:2369] T 0033afea97f447dc99cd5154dc3cdb82 P 845e8192bd174d4e96d4f0a5fdf397f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:01:21.778002 11050 tablet.cc:2369] T 0033afea97f447dc99cd5154dc3cdb82 P 2e51951ff485489b8f1cab74aad4db5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:01:21.779462 11137 raft_consensus.cc:1261] T 0033afea97f447dc99cd5154dc3cdb82 P 845e8192bd174d4e96d4f0a5fdf397f0 [term 1 FOLLOWER]: Refusing update from remote peer 9bce5a795a3d4e8aa276133ca66fcc58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:01:21.779486 11004 raft_consensus.cc:1261] T 0033afea97f447dc99cd5154dc3cdb82 P 2e51951ff485489b8f1cab74aad4db5e [term 1 FOLLOWER]: Refusing update from remote peer 9bce5a795a3d4e8aa276133ca66fcc58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:01:21.780539 11211 consensus_queue.cc:1035] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e51951ff485489b8f1cab74aad4db5e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:01:21.780910 11207 consensus_queue.cc:1035] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "845e8192bd174d4e96d4f0a5fdf397f0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:01:21.788161 10916 tablet.cc:2369] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:01:21.792660 11214 mvcc.cc:205] Tried to move back new op lower bound from 7332972469357772800 to 7332972469155102720. Current Snapshot: MvccSnapshot[applied={T|T < 7332972469357772800}]
I20260924 19:01:21.805352 10724 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:01:21.805670 10724 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:01:21.809207 10724 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6db96d6813d64f448a6aa0f11579888f: Sending DeleteTablet for 3 replicas of tablet 0033afea97f447dc99cd5154dc3cdb82
I20260924 19:01:21.810393 11117 tablet_service.cc:1514] Processing DeleteTablet for tablet 0033afea97f447dc99cd5154dc3cdb82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:01:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40218
I20260924 19:01:21.810750 10850 tablet_service.cc:1514] Processing DeleteTablet for tablet 0033afea97f447dc99cd5154dc3cdb82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:01:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35590
I20260924 19:01:21.811414 10984 tablet_service.cc:1514] Processing DeleteTablet for tablet 0033afea97f447dc99cd5154dc3cdb82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:01:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48352
I20260924 19:01:21.811842 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 10787
I20260924 19:01:21.823683 11225 tablet_replica.cc:330] T 0033afea97f447dc99cd5154dc3cdb82 P 2e51951ff485489b8f1cab74aad4db5e: stopping tablet replica
I20260924 19:01:21.824213 11225 raft_consensus.cc:2229] T 0033afea97f447dc99cd5154dc3cdb82 P 2e51951ff485489b8f1cab74aad4db5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:01:21.824892 11224 tablet_replica.cc:330] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58: stopping tablet replica
I20260924 19:01:21.825453 11225 raft_consensus.cc:2258] T 0033afea97f447dc99cd5154dc3cdb82 P 2e51951ff485489b8f1cab74aad4db5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:01:21.825857 11223 tablet_replica.cc:330] T 0033afea97f447dc99cd5154dc3cdb82 P 845e8192bd174d4e96d4f0a5fdf397f0: stopping tablet replica
I20260924 19:01:21.826260 11224 raft_consensus.cc:2229] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:01:21.826252 11223 raft_consensus.cc:2229] T 0033afea97f447dc99cd5154dc3cdb82 P 845e8192bd174d4e96d4f0a5fdf397f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:01:21.826691 11223 raft_consensus.cc:2258] T 0033afea97f447dc99cd5154dc3cdb82 P 845e8192bd174d4e96d4f0a5fdf397f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:01:21.828330 11225 ts_tablet_manager.cc:1905] T 0033afea97f447dc99cd5154dc3cdb82 P 2e51951ff485489b8f1cab74aad4db5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:01:21.828617 11223 ts_tablet_manager.cc:1905] T 0033afea97f447dc99cd5154dc3cdb82 P 845e8192bd174d4e96d4f0a5fdf397f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:01:21.829061 11224 raft_consensus.cc:2258] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:01:21.831485 11224 ts_tablet_manager.cc:1905] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 19:01:21 dist-test-slave-9fn1 krb5kdc[10680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 19:01:21.836894 11225 ts_tablet_manager.cc:1918] T 0033afea97f447dc99cd5154dc3cdb82 P 2e51951ff485489b8f1cab74aad4db5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:01:21.837144 11225 log.cc:1196] T 0033afea97f447dc99cd5154dc3cdb82 P 2e51951ff485489b8f1cab74aad4db5e: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-1/wal/wals/0033afea97f447dc99cd5154dc3cdb82
I20260924 19:01:21.837818 11225 ts_tablet_manager.cc:1939] T 0033afea97f447dc99cd5154dc3cdb82 P 2e51951ff485489b8f1cab74aad4db5e: Deleting consensus metadata
I20260924 19:01:21.839432 11224 ts_tablet_manager.cc:1918] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:01:21.839587 11224 log.cc:1196] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-0/wal/wals/0033afea97f447dc99cd5154dc3cdb82
I20260924 19:01:21.839802 10711 catalog_manager.cc:4841] TS 2e51951ff485489b8f1cab74aad4db5e (127.0.97.2:45343): tablet 0033afea97f447dc99cd5154dc3cdb82 (table test-table [id=a7e7af552fc5469c9f2f0fa0dc2b9322]) successfully deleted
I20260924 19:01:21.840103 11224 ts_tablet_manager.cc:1939] T 0033afea97f447dc99cd5154dc3cdb82 P 9bce5a795a3d4e8aa276133ca66fcc58: Deleting consensus metadata
I20260924 19:01:21.841735 10711 catalog_manager.cc:4841] TS 9bce5a795a3d4e8aa276133ca66fcc58 (127.0.97.1:41233): tablet 0033afea97f447dc99cd5154dc3cdb82 (table test-table [id=a7e7af552fc5469c9f2f0fa0dc2b9322]) successfully deleted
I20260924 19:01:21.842881 11223 ts_tablet_manager.cc:1918] T 0033afea97f447dc99cd5154dc3cdb82 P 845e8192bd174d4e96d4f0a5fdf397f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:01:21.843040 11223 log.cc:1196] T 0033afea97f447dc99cd5154dc3cdb82 P 845e8192bd174d4e96d4f0a5fdf397f0: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276386554690-388-0/minicluster-data/ts-2/wal/wals/0033afea97f447dc99cd5154dc3cdb82
I20260924 19:01:21.843550 11197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:21.813562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35606 (local address 127.0.97.1:41233)
0924 19:01:21.813718 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:21.813724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:21.822905 (+ 9181us) server_negotiation.cc:407] Connection header received
0924 19:01:21.823373 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:21.823381 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:21.823466 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:21.823574 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:21.825481 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.826667 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.828484 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.829246 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.829343 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:21.834430 (+ 5087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:01:21.834443 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:01:21.834460 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:01:21.834521 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:01:21.838820 (+ 4299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:01:21.840297 (+ 1477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:01:21.840308 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:01:21.840313 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:01:21.840511 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:01:21.841006 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:01:21.841012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:01:21.841015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:01:21.841444 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:01:21.842165 (+ 721us) server_negotiation.cc:1035] Waiting for connection context
0924 19:01:21.842767 (+ 602us) server_negotiation.cc:299] Negotiation successful
0924 19:01:21.843048 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 19:01:21.843559 11223 ts_tablet_manager.cc:1939] T 0033afea97f447dc99cd5154dc3cdb82 P 845e8192bd174d4e96d4f0a5fdf397f0: Deleting consensus metadata
I20260924 19:01:21.844257 10910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:01:21.845124 10709 catalog_manager.cc:4841] TS 845e8192bd174d4e96d4f0a5fdf397f0 (127.0.97.3:46667): tablet 0033afea97f447dc99cd5154dc3cdb82 (table test-table [id=a7e7af552fc5469c9f2f0fa0dc2b9322]) successfully deleted
I20260924 19:01:21.941462 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 10787
I20260924 19:01:21.960433 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 10921
Sep 24 19:01:21 dist-test-slave-9fn1 krb5kdc[10680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 19:01:21.975533 11199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:21.961204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48354 (local address 127.0.97.2:45343)
0924 19:01:21.961369 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:21.961374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:21.961394 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:01:21.961517 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:21.961523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:21.961617 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:21.961717 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:21.962819 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.963997 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.965190 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:21.965831 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:21.965953 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:21.969844 (+ 3891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:01:21.969857 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:01:21.969873 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:01:21.969908 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:01:21.972479 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:01:21.973305 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:01:21.973312 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:01:21.973314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:01:21.973538 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:01:21.973931 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:01:21.973937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:01:21.973940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:01:21.974327 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:01:21.974622 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0924 19:01:21.974993 (+ 371us) server_negotiation.cc:299] Negotiation successful
0924 19:01:21.975206 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 19:01:21.976193 11044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:01:22.064908 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 10921
I20260924 19:01:22.081241 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 11054
Sep 24 19:01:22 dist-test-slave-9fn1 krb5kdc[10680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 19:01:22.096120 11195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:01:22.082060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40220 (local address 127.0.97.3:46667)
0924 19:01:22.082251 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 19:01:22.082256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:01:22.082276 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:01:22.082483 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:01:22.082486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:01:22.082547 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:01:22.082653 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:01:22.083906 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:22.084628 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:22.085997 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:01:22.086399 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:01:22.086543 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:01:22.090577 (+ 4034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:01:22.090590 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:01:22.090606 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:01:22.090642 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:01:22.093867 (+ 3225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:01:22.094449 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:01:22.094456 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:01:22.094459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:01:22.094597 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:01:22.094903 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:01:22.094906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:01:22.094908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:01:22.095258 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:01:22.095459 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 19:01:22.095716 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 19:01:22.095879 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 19:01:22.096810 11177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:01:22.189139 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 11054
I20260924 19:01:22.207886 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 10693
I20260924 19:01:22.208963 10755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:01:22.313393 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 10693
2026-09-24T19:01:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4100 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:01:22 dist-test-slave-9fn1 krb5kdc[11233](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:01:22 dist-test-slave-9fn1 krb5kdc[11233](info): set up 2 sockets
Sep 24 19:01:22 dist-test-slave-9fn1 krb5kdc[11233](info): commencing operation
krb5kdc: starting...
W20260924 19:01:23.402307 388 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:01:23 dist-test-slave-9fn1 krb5kdc[11233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:01:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:01:23Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260924 19:01:23.712025 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskLphABj/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/postgres -l logfile start
2026-09-24 19:01:26.137 UTC [11263] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 19:01:26.137 UTC [11263] LOG: listening on IPv4 address "127.0.97.20", port 46811
2026-09-24 19:01:26.141 UTC [11263] LOG: listening on Unix socket "/tmp/.s.PGSQL.46811"
2026-09-24 19:01:26.163 UTC [11266] LOG: database system was shut down at 2026-09-24 19:01:24 UTC
2026-09-24 19:01:26.167 UTC [11263] LOG: database system is ready to accept connections
I20260924 19:01:27.126796 388 mini_postgres.cc:96] Postgres bound to 46811
2026-09-24 19:01:27.145 UTC [11274] FATAL: database "slave" does not exist
127.0.97.20:46811 - accepting connections
I20260924 19:01:27.146013 388 mini_ranger.cc:161] Starting Ranger...
I20260924 19:01:27.170362 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 19:01:27.636106 388 mini_ranger.cc:87] Created ranger Postgres database
I20260924 19:01:27.636236 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ranger-admin
2026-09-24 19:01:27,940 [I] DB FLAVOR :POSTGRES
2026-09-24 19:01:27,940 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 19:01:27,940 [I] Checking connection..
2026-09-24 19:01:27,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 19:01:28,189 [I] Checking connection passed.
2026-09-24 19:01:28,189 [I] --------- Verifying version history table ---------
2026-09-24 19:01:28,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:01:28,429 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 19:01:28,430 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:01:28,687 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 19:01:28,688 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 19:01:28,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46811/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 19:01:28.941 UTC [11338] WARNING: there is no transaction in progress
2026-09-24 19:01:28,947 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 19:01:28,948 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:01:29,194 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 19:01:29,195 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 19:01:29,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 19:01:29,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 19:01:29,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9fn1.c.gcp-upstream.internal','N') ;"
2026-09-24 19:01:29,918 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 19:01:30,176 [I] Table x_portal_user does not exist in database ranger
2026-09-24 19:01:30,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 19:01:30,424 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 19:01:30,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 19:01:30,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 19:01:30,903 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 19:01:30,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46811/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 19:01:31.810 UTC [11446] WARNING: there is no transaction in progress
2026-09-24 19:01:31.826 UTC [11446] WARNING: there is no transaction in progress
2026-09-24 19:01:31.842 UTC [11446] WARNING: there is no transaction in progress
2026-09-24 19:01:31.856 UTC [11446] WARNING: there is no transaction in progress
2026-09-24 19:01:31.872 UTC [11446] WARNING: there is no transaction in progress
2026-09-24 19:01:31.891 UTC [11446] WARNING: there is no transaction in progress
2026-09-24 19:01:31.945 UTC [11446] WARNING: there is no transaction in progress
2026-09-24 19:01:31.951 UTC [11446] WARNING: there is no transaction in progress
2026-09-24 19:01:31.965 UTC [11446] WARNING: there is no transaction in progress
2026-09-24 19:01:31.975 UTC [11446] WARNING: there is no transaction in progress
2026-09-24 19:01:31.984 UTC [11446] WARNING: there is no transaction in progress
2026-09-24 19:01:31.998 UTC [11446] WARNING: there is no transaction in progress
2026-09-24 19:01:32.007 UTC [11446] WARNING: there is no transaction in progress
2026-09-24 19:01:32.013 UTC [11446] WARNING: there is no transaction in progress
2026-09-24 19:01:32.020 UTC [11446] WARNING: there is no transaction in progress
2026-09-24 19:01:32.026 UTC [11446] WARNING: there is no transaction in progress
2026-09-24 19:01:32,631 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 19:01:32,631 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 19:01:32,879 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 19:01:32,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 19:01:33,104 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 19:01:33,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 19:01:33,366 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 19:01:33,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 19:01:33,591 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 19:01:33,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9fn1.c.gcp-upstream.internal';"
2026-09-24 19:01:34,079 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 19:01:34,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46811/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 19:01:34,301 [I] DB_PATCHES have already been applied
I20260924 19:01:34.307022 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskLphABj/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 19:01:34.307109 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260924 19:01:34.321293 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:34469
Sep 24, 2026 7:01:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:01:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:01:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 7:01:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 7:01:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 7:01:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 7:01:35 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 19:01:35.444655 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.299s user 0.000s sys 0.037s
Sep 24, 2026 7:01:55 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 7:01:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 7:01:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 7:01:55 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 7:01:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 7:01:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 7:01:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 7:01:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 7:01:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 7:01:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 7:01:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 7:01:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 7:01:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 7:01:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 7:01:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 7:01:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 7:01:55 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 7:01:56 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 19:01:57.159996 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 19:01:57 dist-test-slave-9fn1 krb5kdc[11233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276517, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 19:01:57.349486 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 19:01:57.368147 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 19:01:58.197949 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 19:01:58.198119 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ranger-kms
I20260924 19:01:58.198256 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:46811
2026-09-24 19:01:58,624 [I] DB FLAVOR :POSTGRES
2026-09-24 19:01:58,624 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 19:01:58,624 [I] Checking connection
2026-09-24 19:01:58,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46811/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 19:01:58,872 [I] connection success
2026-09-24 19:01:58,872 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 19:01:58,872 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 19:01:58,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46811/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 19:01:59,127 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 19:01:59,127 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 19:01:59,128 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 19:01:59,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:46811/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 19:01:59,372 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 19:01:59.811391 388 mini_ranger_kms.cc:310] Created kms service
I20260924 19:01:59.979060 388 mini_ranger_kms.cc:326] Created kudu user
I20260924 19:02:00.023386 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 19:02:00 dist-test-slave-9fn1 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276517, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 19:02:00.416940 388 mini_ranger_kms.cc:384] Added ranger policy
I20260924 19:02:00.417308 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 19:02:00.417376 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260924 19:02:00.440316 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11626
I20260924 19:02:00.440510 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:51733
Sep 24, 2026 7:02:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:02:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:02:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 7:02:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 7:02:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 7:02:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 19:02:02.714192 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.365s user 0.124s sys 0.319s
I20260924 19:02:02.714435 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 19:02:02.714640 388 mini_ranger_kms.cc:401] 127.0.97.20:51733/kms/v1/keys
Sep 24 19:02:02 dist-test-slave-9fn1 krb5kdc[11233](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790276522, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 19:02:03 dist-test-slave-9fn1 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276522, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 24 19:02:03 dist-test-slave-9fn1 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276522, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 24 19:02:05 dist-test-slave-9fn1 krb5kdc[11233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 19:02:05.893471 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42835
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42835
--ranger_config_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51733/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:02:06.232645 11677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:02:06.232928 11677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:02:06.233075 11677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:02:06.240355 11677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:02:06.240468 11677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:02:06.240551 11677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:02:06.240656 11677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:02:06.240721 11677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:02:06.250901 11677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:51733/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42835
--ranger_config_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:02:06.252173 11677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:02:06.253458 11677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:02:06.266227 11685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:02:06.266259 11683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:02:06.267390 11677 server_base.cc:1034] running on GCE node
W20260924 19:02:06.269779 11682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:02:06.270889 11677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:02:06.272785 11677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:02:06.274037 11677 hybrid_clock.cc:648] HybridClock initialized: now 1790276526274000 us; error 52 us; skew 500 ppm
I20260924 19:02:06.274444 11677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:02:06 dist-test-slave-9fn1 krb5kdc[11233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:02:06.283249 11677 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:02:06.284907 11677 webserver.cc:460] Webserver started at http://127.0.97.62:41013/ using document root <none> and password file <none>
I20260924 19:02:06.285665 11677 fs_manager.cc:359] Metadata directory not provided
I20260924 19:02:06.285789 11677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:02:06.286110 11677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:02:06 dist-test-slave-9fn1 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 19:02:06.350615 11677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "c3f7ef40f8e74175bdbd6213bc5b145c"
format_stamp: "Formatted at 2026-09-24 19:02:06 on dist-test-slave-9fn1"
server_key: "6151b1a88469f8d863c9d7086efb2f36"
server_key_iv: "062c1ff1c4628a39567207cd7c859a48"
server_key_version: "kuduclusterkey@0"
I20260924 19:02:06.351446 11677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "c3f7ef40f8e74175bdbd6213bc5b145c"
format_stamp: "Formatted at 2026-09-24 19:02:06 on dist-test-slave-9fn1"
server_key: "6151b1a88469f8d863c9d7086efb2f36"
server_key_iv: "062c1ff1c4628a39567207cd7c859a48"
server_key_version: "kuduclusterkey@0"
I20260924 19:02:06.356925 11677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:02:06.373358 11692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:02:06.374981 11677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:02:06.375164 11677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "c3f7ef40f8e74175bdbd6213bc5b145c"
format_stamp: "Formatted at 2026-09-24 19:02:06 on dist-test-slave-9fn1"
server_key: "6151b1a88469f8d863c9d7086efb2f36"
server_key_iv: "062c1ff1c4628a39567207cd7c859a48"
server_key_version: "kuduclusterkey@0"
I20260924 19:02:06.375363 11677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:02:06.403322 11677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:02:06.407367 11677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:02:06.407691 11677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:02:06.428846 11677 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42835
I20260924 19:02:06.428849 11743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42835 every 8 connection(s)
I20260924 19:02:06.430255 11677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:02:06.434154 11744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:02:06.437126 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 11677
I20260924 19:02:06.437319 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:02:06.438107 388 external_mini_cluster.cc:1428] Setting key 4b7b9b82ae43d2f249e3fd2244d1051c
I20260924 19:02:06.449301 11744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f7ef40f8e74175bdbd6213bc5b145c: Bootstrap starting.
I20260924 19:02:06.454949 11744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3f7ef40f8e74175bdbd6213bc5b145c: Neither blocks nor log segments found. Creating new log.
I20260924 19:02:06.456421 11744 log.cc:824] T 00000000000000000000000000000000 P c3f7ef40f8e74175bdbd6213bc5b145c: Log is configured to *not* fsync() on all Append() calls
I20260924 19:02:06.460175 11744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3f7ef40f8e74175bdbd6213bc5b145c: No bootstrap required, opened a new log
Sep 24 19:02:06 dist-test-slave-9fn1 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:02:06.469439 11744 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3f7ef40f8e74175bdbd6213bc5b145c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f7ef40f8e74175bdbd6213bc5b145c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42835 } }
I20260924 19:02:06.469969 11744 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3f7ef40f8e74175bdbd6213bc5b145c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:02:06.470077 11744 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3f7ef40f8e74175bdbd6213bc5b145c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3f7ef40f8e74175bdbd6213bc5b145c, State: Initialized, Role: FOLLOWER
I20260924 19:02:06.470845 11744 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3f7ef40f8e74175bdbd6213bc5b145c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f7ef40f8e74175bdbd6213bc5b145c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42835 } }
I20260924 19:02:06.471133 11744 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3f7ef40f8e74175bdbd6213bc5b145c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:02:06.471319 11744 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3f7ef40f8e74175bdbd6213bc5b145c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:02:06.471508 11744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3f7ef40f8e74175bdbd6213bc5b145c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:02:06.474116 11744 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3f7ef40f8e74175bdbd6213bc5b145c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f7ef40f8e74175bdbd6213bc5b145c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42835 } }
I20260924 19:02:06.474689 11744 leader_election.cc:302] T 00000000000000000000000000000000 P c3f7ef40f8e74175bdbd6213bc5b145c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3f7ef40f8e74175bdbd6213bc5b145c; no voters:
I20260924 19:02:06.475397 11744 leader_election.cc:288] T 00000000000000000000000000000000 P c3f7ef40f8e74175bdbd6213bc5b145c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:02:06.475492 11749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3f7ef40f8e74175bdbd6213bc5b145c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:02:06.480686 11747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:06.440229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49684 (local address 127.0.97.62:42835)
0924 19:02:06.455322 (+ 15093us) server_negotiation.cc:206] Beginning negotiation
0924 19:02:06.455340 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 19:02:06.455398 (+ 58us) server_negotiation.cc:407] Connection header received
0924 19:02:06.457806 (+ 2408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:02:06.457833 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:02:06.458318 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:02:06.458730 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:02:06.460782 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:06.462853 (+ 2071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:06.464542 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:06.465668 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:06.466248 (+ 580us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:06.470975 (+ 4727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:02:06.470985 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:02:06.471009 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:02:06.471052 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:02:06.474703 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:06.475594 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:06.475603 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:06.475611 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:06.475791 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:06.476366 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:06.476373 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:06.476379 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:06.477223 (+ 844us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:02:06.477589 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0924 19:02:06.478485 (+ 896us) server_negotiation.cc:299] Negotiation successful
0924 19:02:06.479004 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14840,"thread_start_us":14644,"threads_started":1}
I20260924 19:02:06.483703 11749 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3f7ef40f8e74175bdbd6213bc5b145c [term 1 LEADER]: Becoming Leader. State: Replica: c3f7ef40f8e74175bdbd6213bc5b145c, State: Running, Role: LEADER
I20260924 19:02:06.484344 11749 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3f7ef40f8e74175bdbd6213bc5b145c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f7ef40f8e74175bdbd6213bc5b145c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42835 } }
I20260924 19:02:06.484872 11744 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3f7ef40f8e74175bdbd6213bc5b145c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:02:06.487512 11750 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f7ef40f8e74175bdbd6213bc5b145c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3f7ef40f8e74175bdbd6213bc5b145c. Latest consensus state: current_term: 1 leader_uuid: "c3f7ef40f8e74175bdbd6213bc5b145c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f7ef40f8e74175bdbd6213bc5b145c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42835 } } }
I20260924 19:02:06.487850 11750 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f7ef40f8e74175bdbd6213bc5b145c [sys.catalog]: This master's current role is: LEADER
I20260924 19:02:06.491533 11751 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3f7ef40f8e74175bdbd6213bc5b145c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3f7ef40f8e74175bdbd6213bc5b145c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3f7ef40f8e74175bdbd6213bc5b145c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42835 } } }
I20260924 19:02:06.492273 11751 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3f7ef40f8e74175bdbd6213bc5b145c [sys.catalog]: This master's current role is: LEADER
I20260924 19:02:06.498248 11754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:02:06.506548 11754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:02:06.524645 11754 catalog_manager.cc:1347] Generated new cluster ID: 3266b79185d64ee5928f2da758a8541d
I20260924 19:02:06.525346 11754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:02:06.537760 11754 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:02:06.541992 11754 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:02:06.558452 11754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3f7ef40f8e74175bdbd6213bc5b145c: Generated new TSK 0
I20260924 19:02:06.559906 11754 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:02:06.563504 11744 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 19:02:07.336946 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51733/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42835
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:02:07 dist-test-slave-9fn1 krb5kdc[11233](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790276527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 19:02:07.755281 11789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:02:07.755688 11789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:02:07.755905 11789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:02:07.767771 11789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:02:07.767976 11789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:02:07.768205 11789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:02:07.784430 11789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:51733/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:02:07.786222 11789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:02:07.787937 11789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:02:07.808139 11800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:02:07.809732 11799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:02:07.814272 11789 server_base.cc:1034] running on GCE node
W20260924 19:02:07.814558 11802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:02:07.815836 11789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:02:07.816982 11789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:02:07.818286 11789 hybrid_clock.cc:648] HybridClock initialized: now 1790276527818216 us; error 59 us; skew 500 ppm
I20260924 19:02:07.818651 11789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:02:07 dist-test-slave-9fn1 krb5kdc[11233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:02:07.822409 11789 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 19:02:07.825099 11789 webserver.cc:460] Webserver started at http://127.0.97.1:41835/ using document root <none> and password file <none>
I20260924 19:02:07.826057 11789 fs_manager.cc:359] Metadata directory not provided
I20260924 19:02:07.826233 11789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:02:07.826615 11789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:02:07 dist-test-slave-9fn1 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 19:02:07.853485 11789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-0/data/instance:
uuid: "0e5286592eb5486cb8b93d1bdc6d98da"
format_stamp: "Formatted at 2026-09-24 19:02:07 on dist-test-slave-9fn1"
server_key: "25b57d5e9ec9dec6c7311c230367cb0e"
server_key_iv: "6552fa07599fcb349c2c80c0e3109f43"
server_key_version: "kuduclusterkey@0"
I20260924 19:02:07.854761 11789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0e5286592eb5486cb8b93d1bdc6d98da"
format_stamp: "Formatted at 2026-09-24 19:02:07 on dist-test-slave-9fn1"
server_key: "25b57d5e9ec9dec6c7311c230367cb0e"
server_key_iv: "6552fa07599fcb349c2c80c0e3109f43"
server_key_version: "kuduclusterkey@0"
I20260924 19:02:07.868284 11789 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.011s sys 0.000s
I20260924 19:02:07.899189 11809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:02:07.901132 11789 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.004s sys 0.004s
I20260924 19:02:07.901542 11789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "0e5286592eb5486cb8b93d1bdc6d98da"
format_stamp: "Formatted at 2026-09-24 19:02:07 on dist-test-slave-9fn1"
server_key: "25b57d5e9ec9dec6c7311c230367cb0e"
server_key_iv: "6552fa07599fcb349c2c80c0e3109f43"
server_key_version: "kuduclusterkey@0"
I20260924 19:02:07.901913 11789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:02:07.963521 11789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:02:07.969866 11789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:02:07.970233 11789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:02:07.971561 11789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:02:07.973667 11789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:02:07.973770 11789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:02:07.973912 11789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:02:07.973987 11789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:02:08.048249 11789 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38365
I20260924 19:02:08.048292 11921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38365 every 8 connection(s)
I20260924 19:02:08.049947 11789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:02:08.056432 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 11789
I20260924 19:02:08.056633 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-0/wal/instance
I20260924 19:02:08.057440 388 external_mini_cluster.cc:1428] Setting key 0f9f5774b4e3f4eced1b3609294de124
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Sep 24 19:02:08 dist-test-slave-9fn1 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260924 19:02:08.097640 11925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:08.055933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42835 (local address 127.0.97.1:48393)
0924 19:02:08.067185 (+ 11252us) negotiation.cc:107] Waiting for socket to connect
0924 19:02:08.067251 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 19:02:08.071877 (+ 4626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:02:08.076702 (+ 4825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:02:08.076719 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:02:08.077406 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:02:08.078294 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:02:08.078324 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:02:08.080078 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:02:08.080086 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:02:08.082432 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:02:08.082457 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:02:08.083996 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:02:08.084006 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:02:08.084334 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:08.085193 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:02:08.085227 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:02:08.088090 (+ 2863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:02:08.092722 (+ 4632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:02:08.092732 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:02:08.092763 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:02:08.093392 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:02:08.093864 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:02:08.093870 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:02:08.093874 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:02:08.094121 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:02:08.094819 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:02:08.094828 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:02:08.095300 (+ 472us) client_negotiation.cc:769] Sending connection context
0924 19:02:08.095610 (+ 310us) client_negotiation.cc:240] Negotiation successful
0924 19:02:08.096178 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10797,"thread_start_us":13416,"threads_started":1}
I20260924 19:02:08.098788 11924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:08.055206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48393 (local address 127.0.97.62:42835)
0924 19:02:08.070907 (+ 15701us) server_negotiation.cc:206] Beginning negotiation
0924 19:02:08.070916 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:02:08.070942 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:02:08.072207 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:02:08.072212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:02:08.072316 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:02:08.072421 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:02:08.078536 (+ 6115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:08.079830 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:08.082638 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:08.083466 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:08.083604 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:08.088342 (+ 4738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:02:08.088347 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:02:08.088350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:02:08.088382 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:02:08.092512 (+ 4130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:08.093568 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:08.093573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:08.093576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:08.093725 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:08.094266 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:08.094271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:08.094275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:08.094608 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:02:08.094748 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 19:02:08.098176 (+ 3428us) server_negotiation.cc:299] Negotiation successful
0924 19:02:08.098406 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15561,"thread_start_us":15453,"threads_started":1}
I20260924 19:02:08.101280 11922 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42835
I20260924 19:02:08.101624 11922 heartbeater.cc:464] Registering TS with master...
I20260924 19:02:08.102504 11922 heartbeater.cc:511] Master 127.0.97.62:42835 requested a full tablet report, sending...
I20260924 19:02:08.105144 11709 ts_manager.cc:194] Registered new tserver with Master: 0e5286592eb5486cb8b93d1bdc6d98da (127.0.97.1:38365)
I20260924 19:02:08.107280 11709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 19:02:08.124895 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51733/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42835
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:02:08 dist-test-slave-9fn1 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 24 19:02:08 dist-test-slave-9fn1 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260924 19:02:08.541818 11928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:02:08.542208 11928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:02:08.542860 11928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:02:08.553041 11928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:02:08.553194 11928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:02:08.553404 11928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:02:08.565752 11928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:51733/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:02:08.567725 11928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:02:08.569605 11928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:02:08.641036 11935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:02:08.641709 11936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:02:08.658793 11938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:02:08.660210 11928 server_base.cc:1034] running on GCE node
I20260924 19:02:08.661082 11928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:02:08.675221 11928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:02:08.676568 11928 hybrid_clock.cc:648] HybridClock initialized: now 1790276528676472 us; error 90 us; skew 500 ppm
I20260924 19:02:08.677048 11928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:02:08 dist-test-slave-9fn1 krb5kdc[11233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:02:08.681272 11928 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 19:02:08.684466 11928 webserver.cc:460] Webserver started at http://127.0.97.2:41015/ using document root <none> and password file <none>
I20260924 19:02:08.685443 11928 fs_manager.cc:359] Metadata directory not provided
I20260924 19:02:08.685653 11928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:02:08.686146 11928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:02:08 dist-test-slave-9fn1 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 19:02:08.705492 11928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-1/data/instance:
uuid: "880376e464394ea39516002498a4fe51"
format_stamp: "Formatted at 2026-09-24 19:02:08 on dist-test-slave-9fn1"
server_key: "f486b917e154e9511395881c4b055404"
server_key_iv: "2abe5582349c873ea55d95af2c303cf5"
server_key_version: "kuduclusterkey@0"
I20260924 19:02:08.706825 11928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-1/wal/instance:
uuid: "880376e464394ea39516002498a4fe51"
format_stamp: "Formatted at 2026-09-24 19:02:08 on dist-test-slave-9fn1"
server_key: "f486b917e154e9511395881c4b055404"
server_key_iv: "2abe5582349c873ea55d95af2c303cf5"
server_key_version: "kuduclusterkey@0"
I20260924 19:02:08.722951 11928 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.010s sys 0.005s
I20260924 19:02:08.761762 11945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:02:08.763684 11928 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.005s sys 0.005s
I20260924 19:02:08.763971 11928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "880376e464394ea39516002498a4fe51"
format_stamp: "Formatted at 2026-09-24 19:02:08 on dist-test-slave-9fn1"
server_key: "f486b917e154e9511395881c4b055404"
server_key_iv: "2abe5582349c873ea55d95af2c303cf5"
server_key_version: "kuduclusterkey@0"
I20260924 19:02:08.764262 11928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:02:08.814278 11928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:02:08.824172 11928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:02:08.824576 11928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:02:08.837972 11928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:02:08.851146 11928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:02:08.851267 11928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:02:08.851464 11928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:02:08.851536 11928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:02:08.896096 11928 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44285
I20260924 19:02:08.896154 12064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44285 every 8 connection(s)
I20260924 19:02:08.897872 11928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:02:08.901846 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 11928
I20260924 19:02:08.902022 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-1/wal/instance
I20260924 19:02:08.902565 388 external_mini_cluster.cc:1428] Setting key deac933dcb7ec37b39bfa236612f7e2e
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Sep 24 19:02:08 dist-test-slave-9fn1 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:02:08.941643 12067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:08.900703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54515 (local address 127.0.97.62:42835)
0924 19:02:08.903318 (+ 2615us) server_negotiation.cc:206] Beginning negotiation
0924 19:02:08.903325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:02:08.918084 (+ 14759us) server_negotiation.cc:407] Connection header received
0924 19:02:08.920828 (+ 2744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:02:08.920835 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:02:08.920944 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:02:08.921026 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:02:08.923414 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:08.924591 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:08.927097 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:08.927737 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:08.927859 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:08.934224 (+ 6365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:02:08.934230 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:02:08.934235 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:02:08.934272 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:02:08.937581 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:08.938921 (+ 1340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:08.938927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:08.938930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:08.939073 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:08.939566 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:08.939571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:08.939574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:08.939918 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:02:08.940039 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 19:02:08.941031 (+ 992us) server_negotiation.cc:299] Negotiation successful
0924 19:02:08.941251 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":625,"threads_started":1}
I20260924 19:02:08.944036 12068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:08.901863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42835 (local address 127.0.97.2:54515)
0924 19:02:08.916898 (+ 15035us) negotiation.cc:107] Waiting for socket to connect
0924 19:02:08.916964 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 19:02:08.920448 (+ 3484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:02:08.921625 (+ 1177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:02:08.921637 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:02:08.922327 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:02:08.923211 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:02:08.923236 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:02:08.924808 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:02:08.924813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:02:08.926924 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:02:08.926947 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:02:08.928179 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:02:08.928186 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:02:08.928474 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:08.929297 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:02:08.929328 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:02:08.933979 (+ 4651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:02:08.937738 (+ 3759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:02:08.937747 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:02:08.937767 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:02:08.938774 (+ 1007us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:02:08.939203 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:02:08.939207 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:02:08.939210 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:02:08.939443 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:02:08.940125 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:02:08.940133 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:02:08.940583 (+ 450us) client_negotiation.cc:769] Sending connection context
0924 19:02:08.941998 (+ 1415us) client_negotiation.cc:240] Negotiation successful
0924 19:02:08.942542 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14648,"thread_start_us":15084,"threads_started":1}
I20260924 19:02:08.945791 12065 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42835
I20260924 19:02:08.946100 12065 heartbeater.cc:464] Registering TS with master...
I20260924 19:02:08.946962 12065 heartbeater.cc:511] Master 127.0.97.62:42835 requested a full tablet report, sending...
I20260924 19:02:08.948702 11709 ts_manager.cc:194] Registered new tserver with Master: 880376e464394ea39516002498a4fe51 (127.0.97.2:44285)
I20260924 19:02:08.949846 11709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 19:02:08.966451 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:51733/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42835
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:02:09.111568 11922 heartbeater.cc:503] Master 127.0.97.62:42835 was elected leader, sending a full tablet report...
W20260924 19:02:09.329331 12071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:02:09.329698 12071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:02:09.329864 12071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:02:09.337067 12071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:02:09.337702 12071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:02:09.337883 12071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:02:09.348771 12071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:51733/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:02:09.350142 12071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:02:09.351315 12071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:02:09.364326 12076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:02:09.364225 12077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:02:09.364463 12079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:02:09.365396 12071 server_base.cc:1034] running on GCE node
I20260924 19:02:09.366031 12071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:02:09.366956 12071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:02:09.368180 12071 hybrid_clock.cc:648] HybridClock initialized: now 1790276529368133 us; error 40 us; skew 500 ppm
I20260924 19:02:09.368520 12071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:02:09 dist-test-slave-9fn1 krb5kdc[11233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:02:09.371731 12071 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 19:02:09.373222 12071 webserver.cc:460] Webserver started at http://127.0.97.3:33427/ using document root <none> and password file <none>
I20260924 19:02:09.373961 12071 fs_manager.cc:359] Metadata directory not provided
I20260924 19:02:09.374104 12071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:02:09.374420 12071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:02:09 dist-test-slave-9fn1 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 19:02:09.388885 12071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-2/data/instance:
uuid: "b47cba426e604b94811d61195672a6a1"
format_stamp: "Formatted at 2026-09-24 19:02:09 on dist-test-slave-9fn1"
server_key: "306aaa26a44ebf41b01fcda126605250"
server_key_iv: "e3aa25587680b606f2280d0823f41e1d"
server_key_version: "kuduclusterkey@0"
I20260924 19:02:09.389806 12071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b47cba426e604b94811d61195672a6a1"
format_stamp: "Formatted at 2026-09-24 19:02:09 on dist-test-slave-9fn1"
server_key: "306aaa26a44ebf41b01fcda126605250"
server_key_iv: "e3aa25587680b606f2280d0823f41e1d"
server_key_version: "kuduclusterkey@0"
I20260924 19:02:09.395607 12071 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 19:02:09.411932 12086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:02:09.413394 12071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260924 19:02:09.413623 12071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "b47cba426e604b94811d61195672a6a1"
format_stamp: "Formatted at 2026-09-24 19:02:09 on dist-test-slave-9fn1"
server_key: "306aaa26a44ebf41b01fcda126605250"
server_key_iv: "e3aa25587680b606f2280d0823f41e1d"
server_key_version: "kuduclusterkey@0"
I20260924 19:02:09.413864 12071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:02:09.427510 12071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:02:09.431677 12071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:02:09.431960 12071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:02:09.433329 12071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:02:09.434777 12071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:02:09.434850 12071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:02:09.435017 12071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:02:09.435070 12071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:02:09.470080 12071 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37629
I20260924 19:02:09.470098 12198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37629 every 8 connection(s)
I20260924 19:02:09.471422 12071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:02:09.475466 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 12071
I20260924 19:02:09.475643 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-2/wal/instance
I20260924 19:02:09.476099 388 external_mini_cluster.cc:1428] Setting key 1a40800c8e64956b9a35e78b0c4a787a
Sep 24 19:02:09 dist-test-slave-9fn1 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:02:09.490991 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:09.473839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50609 (local address 127.0.97.62:42835)
0924 19:02:09.474305 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0924 19:02:09.474310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:02:09.475096 (+ 786us) server_negotiation.cc:407] Connection header received
0924 19:02:09.477026 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:02:09.477032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:02:09.477140 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:02:09.477253 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:02:09.479157 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.479919 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.481433 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.481981 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.482122 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:09.485325 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:02:09.485330 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:02:09.485333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:02:09.485364 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:02:09.487865 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:09.488579 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:09.488585 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:09.488589 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:09.488711 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:09.489183 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:09.489189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:09.489192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:09.489463 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:02:09.489635 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 19:02:09.490399 (+ 764us) server_negotiation.cc:299] Negotiation successful
0924 19:02:09.490610 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":252,"threads_started":1}
I20260924 19:02:09.491968 12202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:09.474279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42835 (local address 127.0.97.3:50609)
0924 19:02:09.474953 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0924 19:02:09.474988 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:02:09.476753 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:02:09.477492 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:02:09.477543 (+ 51us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:02:09.477956 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:02:09.478998 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:02:09.479014 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.480068 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.480072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:02:09.481268 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:02:09.481286 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.482218 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.482224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:02:09.482465 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:09.483088 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:02:09.483109 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:02:09.485091 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:02:09.488009 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:02:09.488018 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:02:09.488034 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:02:09.488460 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:02:09.488869 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:02:09.488876 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:02:09.488879 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:02:09.489073 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:02:09.489655 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:02:09.489663 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:02:09.490005 (+ 342us) client_negotiation.cc:769] Sending connection context
0924 19:02:09.490309 (+ 304us) client_negotiation.cc:240] Negotiation successful
0924 19:02:09.490777 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":300,"threads_started":1}
I20260924 19:02:09.493194 12199 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42835
I20260924 19:02:09.493544 12199 heartbeater.cc:464] Registering TS with master...
I20260924 19:02:09.494234 12199 heartbeater.cc:511] Master 127.0.97.62:42835 requested a full tablet report, sending...
I20260924 19:02:09.495572 11709 ts_manager.cc:194] Registered new tserver with Master: b47cba426e604b94811d61195672a6a1 (127.0.97.3:37629)
I20260924 19:02:09.496347 11709 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50609
I20260924 19:02:09.503825 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:02:09.521134 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:09.508452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49690 (local address 127.0.97.62:42835)
0924 19:02:09.508935 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0924 19:02:09.508941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:02:09.509565 (+ 624us) server_negotiation.cc:407] Connection header received
0924 19:02:09.509738 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:02:09.509742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:02:09.509813 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:02:09.509909 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:02:09.511255 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.511969 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.513257 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.513758 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.513862 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:09.516262 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:02:09.516266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:02:09.516269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:02:09.516302 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:02:09.518983 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:09.519572 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:09.519575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:09.519578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:09.519684 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:09.519993 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:09.519996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:09.519997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:09.520236 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:02:09.520404 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 19:02:09.520665 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 19:02:09.520821 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343}
I20260924 19:02:09.524938 11709 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:02:09.532406 11709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:02:09.568384 12212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:09.544460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37629 (local address 127.0.0.1:53944)
0924 19:02:09.545462 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0924 19:02:09.545531 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 19:02:09.545686 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:02:09.559288 (+ 13602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:02:09.559295 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:02:09.559338 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:02:09.559882 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:02:09.559898 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.562473 (+ 2575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.562480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:02:09.565123 (+ 2643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:02:09.565134 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.567028 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.567037 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:02:09.567365 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:09.567417 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 19:02:09.567570 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 19:02:09.567759 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":49,"thread_start_us":674,"threads_started":1}
I20260924 19:02:09.568521 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:09.544876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53944 (local address 127.0.97.3:37629)
0924 19:02:09.556694 (+ 11818us) server_negotiation.cc:206] Beginning negotiation
0924 19:02:09.556706 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:02:09.556752 (+ 46us) server_negotiation.cc:407] Connection header received
0924 19:02:09.556884 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:02:09.556895 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:02:09.557243 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:02:09.560168 (+ 2925us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:02:09.560869 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.562336 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.565323 (+ 2987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.566807 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.566935 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:09.567158 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 19:02:09.568008 (+ 850us) server_negotiation.cc:299] Negotiation successful
0924 19:02:09.568170 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11581,"thread_start_us":11413,"threads_started":1}
I20260924 19:02:09.568847 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:09.544802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44285 (local address 127.0.0.1:37806)
0924 19:02:09.557973 (+ 13171us) negotiation.cc:107] Waiting for socket to connect
0924 19:02:09.557999 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:02:09.558140 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:02:09.562409 (+ 4269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:02:09.562414 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:02:09.562445 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:02:09.562917 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:02:09.562934 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.564309 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.564315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:02:09.566179 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:02:09.566190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.568062 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.568071 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:02:09.568254 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:09.568284 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 19:02:09.568389 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 19:02:09.568524 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13009,"mutex_wait_us":4,"thread_start_us":13458,"threads_started":1}
I20260924 19:02:09.570204 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:09.545160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37806 (local address 127.0.97.2:44285)
0924 19:02:09.561017 (+ 15857us) server_negotiation.cc:206] Beginning negotiation
0924 19:02:09.561025 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:02:09.561054 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:02:09.561143 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:02:09.561150 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:02:09.562117 (+ 967us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:02:09.562301 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:02:09.563112 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.564176 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.566334 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.567514 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.567624 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:09.567784 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 19:02:09.569385 (+ 1601us) server_negotiation.cc:299] Negotiation successful
0924 19:02:09.569573 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15679,"thread_start_us":16695,"threads_started":1}
I20260924 19:02:09.570601 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:09.545870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38365 (local address 127.0.0.1:41730)
0924 19:02:09.561048 (+ 15178us) negotiation.cc:107] Waiting for socket to connect
0924 19:02:09.561072 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:02:09.561212 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:02:09.562607 (+ 1395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:02:09.562615 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:02:09.562645 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:02:09.563142 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:02:09.563155 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.565134 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.565139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:02:09.567337 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:02:09.567348 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.569860 (+ 2512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.569866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:02:09.570039 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:09.570079 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 19:02:09.570160 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 19:02:09.570311 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15013,"mutex_wait_us":78,"thread_start_us":15439,"threads_started":1}
I20260924 19:02:09.571379 12217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:09.546207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41730 (local address 127.0.97.1:38365)
0924 19:02:09.561963 (+ 15756us) server_negotiation.cc:206] Beginning negotiation
0924 19:02:09.561971 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:02:09.561998 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:02:09.562086 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:02:09.562093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:02:09.562318 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:02:09.562502 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:02:09.563348 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.564910 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.567489 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.569369 (+ 1880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.569487 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:09.569709 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 19:02:09.570930 (+ 1221us) server_negotiation.cc:299] Negotiation successful
0924 19:02:09.571110 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15591,"thread_start_us":15437,"threads_started":1}
I20260924 19:02:09.572860 12134 tablet_service.cc:1467] Processing CreateTablet for tablet dea5897497344d15b64488945a46ce9f (DEFAULT_TABLE table=test-table [id=06bb2af41dd94aaea94464bfc1bd1d39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:02:09.573623 12000 tablet_service.cc:1467] Processing CreateTablet for tablet dea5897497344d15b64488945a46ce9f (DEFAULT_TABLE table=test-table [id=06bb2af41dd94aaea94464bfc1bd1d39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:02:09.574165 12134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dea5897497344d15b64488945a46ce9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:02:09.574905 12000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dea5897497344d15b64488945a46ce9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:02:09.574879 11857 tablet_service.cc:1467] Processing CreateTablet for tablet dea5897497344d15b64488945a46ce9f (DEFAULT_TABLE table=test-table [id=06bb2af41dd94aaea94464bfc1bd1d39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:02:09.576225 11857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dea5897497344d15b64488945a46ce9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:02:09.585361 12218 tablet_bootstrap.cc:492] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1: Bootstrap starting.
I20260924 19:02:09.588897 12218 tablet_bootstrap.cc:654] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1: Neither blocks nor log segments found. Creating new log.
I20260924 19:02:09.590193 12219 tablet_bootstrap.cc:492] T dea5897497344d15b64488945a46ce9f P 880376e464394ea39516002498a4fe51: Bootstrap starting.
I20260924 19:02:09.590185 12218 log.cc:824] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1: Log is configured to *not* fsync() on all Append() calls
I20260924 19:02:09.593611 12220 tablet_bootstrap.cc:492] T dea5897497344d15b64488945a46ce9f P 0e5286592eb5486cb8b93d1bdc6d98da: Bootstrap starting.
I20260924 19:02:09.593763 12218 tablet_bootstrap.cc:492] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1: No bootstrap required, opened a new log
I20260924 19:02:09.593812 12219 tablet_bootstrap.cc:654] T dea5897497344d15b64488945a46ce9f P 880376e464394ea39516002498a4fe51: Neither blocks nor log segments found. Creating new log.
I20260924 19:02:09.594039 12218 ts_tablet_manager.cc:1397] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 19:02:09.595387 12219 log.cc:824] T dea5897497344d15b64488945a46ce9f P 880376e464394ea39516002498a4fe51: Log is configured to *not* fsync() on all Append() calls
I20260924 19:02:09.596220 12220 tablet_bootstrap.cc:654] T dea5897497344d15b64488945a46ce9f P 0e5286592eb5486cb8b93d1bdc6d98da: Neither blocks nor log segments found. Creating new log.
I20260924 19:02:09.597291 12220 log.cc:824] T dea5897497344d15b64488945a46ce9f P 0e5286592eb5486cb8b93d1bdc6d98da: Log is configured to *not* fsync() on all Append() calls
I20260924 19:02:09.598990 12219 tablet_bootstrap.cc:492] T dea5897497344d15b64488945a46ce9f P 880376e464394ea39516002498a4fe51: No bootstrap required, opened a new log
I20260924 19:02:09.599233 12219 ts_tablet_manager.cc:1397] T dea5897497344d15b64488945a46ce9f P 880376e464394ea39516002498a4fe51: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 19:02:09.600193 12220 tablet_bootstrap.cc:492] T dea5897497344d15b64488945a46ce9f P 0e5286592eb5486cb8b93d1bdc6d98da: No bootstrap required, opened a new log
I20260924 19:02:09.600443 12220 ts_tablet_manager.cc:1397] T dea5897497344d15b64488945a46ce9f P 0e5286592eb5486cb8b93d1bdc6d98da: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260924 19:02:09.604401 12218 raft_consensus.cc:348] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b47cba426e604b94811d61195672a6a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37629 } } peers { permanent_uuid: "880376e464394ea39516002498a4fe51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44285 } } peers { permanent_uuid: "0e5286592eb5486cb8b93d1bdc6d98da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } }
I20260924 19:02:09.605237 12218 raft_consensus.cc:374] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:02:09.605337 12218 raft_consensus.cc:729] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b47cba426e604b94811d61195672a6a1, State: Initialized, Role: FOLLOWER
I20260924 19:02:09.605960 12218 consensus_queue.cc:260] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b47cba426e604b94811d61195672a6a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37629 } } peers { permanent_uuid: "880376e464394ea39516002498a4fe51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44285 } } peers { permanent_uuid: "0e5286592eb5486cb8b93d1bdc6d98da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } }
I20260924 19:02:09.607969 12220 raft_consensus.cc:348] T dea5897497344d15b64488945a46ce9f P 0e5286592eb5486cb8b93d1bdc6d98da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b47cba426e604b94811d61195672a6a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37629 } } peers { permanent_uuid: "880376e464394ea39516002498a4fe51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44285 } } peers { permanent_uuid: "0e5286592eb5486cb8b93d1bdc6d98da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } }
I20260924 19:02:09.608434 12220 raft_consensus.cc:374] T dea5897497344d15b64488945a46ce9f P 0e5286592eb5486cb8b93d1bdc6d98da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:02:09.608573 12220 raft_consensus.cc:729] T dea5897497344d15b64488945a46ce9f P 0e5286592eb5486cb8b93d1bdc6d98da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e5286592eb5486cb8b93d1bdc6d98da, State: Initialized, Role: FOLLOWER
I20260924 19:02:09.609228 12220 consensus_queue.cc:260] T dea5897497344d15b64488945a46ce9f P 0e5286592eb5486cb8b93d1bdc6d98da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b47cba426e604b94811d61195672a6a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37629 } } peers { permanent_uuid: "880376e464394ea39516002498a4fe51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44285 } } peers { permanent_uuid: "0e5286592eb5486cb8b93d1bdc6d98da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } }
I20260924 19:02:09.610970 12220 ts_tablet_manager.cc:1428] T dea5897497344d15b64488945a46ce9f P 0e5286592eb5486cb8b93d1bdc6d98da: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260924 19:02:09.610719 12219 raft_consensus.cc:348] T dea5897497344d15b64488945a46ce9f P 880376e464394ea39516002498a4fe51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b47cba426e604b94811d61195672a6a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37629 } } peers { permanent_uuid: "880376e464394ea39516002498a4fe51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44285 } } peers { permanent_uuid: "0e5286592eb5486cb8b93d1bdc6d98da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } }
I20260924 19:02:09.611318 12219 raft_consensus.cc:374] T dea5897497344d15b64488945a46ce9f P 880376e464394ea39516002498a4fe51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:02:09.611493 12219 raft_consensus.cc:729] T dea5897497344d15b64488945a46ce9f P 880376e464394ea39516002498a4fe51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 880376e464394ea39516002498a4fe51, State: Initialized, Role: FOLLOWER
I20260924 19:02:09.612207 12219 consensus_queue.cc:260] T dea5897497344d15b64488945a46ce9f P 880376e464394ea39516002498a4fe51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b47cba426e604b94811d61195672a6a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37629 } } peers { permanent_uuid: "880376e464394ea39516002498a4fe51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44285 } } peers { permanent_uuid: "0e5286592eb5486cb8b93d1bdc6d98da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } }
I20260924 19:02:09.613346 12065 heartbeater.cc:503] Master 127.0.97.62:42835 was elected leader, sending a full tablet report...
I20260924 19:02:09.613732 12199 heartbeater.cc:503] Master 127.0.97.62:42835 was elected leader, sending a full tablet report...
I20260924 19:02:09.615296 12218 ts_tablet_manager.cc:1428] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1: Time spent starting tablet: real 0.021s user 0.015s sys 0.004s
I20260924 19:02:09.616789 12219 ts_tablet_manager.cc:1428] T dea5897497344d15b64488945a46ce9f P 880376e464394ea39516002498a4fe51: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
W20260924 19:02:09.651206 12066 tablet.cc:2369] T dea5897497344d15b64488945a46ce9f P 880376e464394ea39516002498a4fe51: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:02:09.722972 12200 tablet.cc:2369] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:02:09.800405 12224 raft_consensus.cc:482] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:02:09.800720 12224 raft_consensus.cc:504] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b47cba426e604b94811d61195672a6a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37629 } } peers { permanent_uuid: "880376e464394ea39516002498a4fe51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44285 } } peers { permanent_uuid: "0e5286592eb5486cb8b93d1bdc6d98da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } }
I20260924 19:02:09.802482 12224 leader_election.cc:288] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 880376e464394ea39516002498a4fe51 (127.0.97.2:44285), 0e5286592eb5486cb8b93d1bdc6d98da (127.0.97.1:38365)
W20260924 19:02:09.805388 11923 tablet.cc:2369] T dea5897497344d15b64488945a46ce9f P 0e5286592eb5486cb8b93d1bdc6d98da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:02:09.808434 12202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:09.802731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38365 (local address 127.0.97.3:41625)
0924 19:02:09.803011 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 19:02:09.803034 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:02:09.803150 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:02:09.803482 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:02:09.803486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:02:09.803510 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:02:09.803788 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:02:09.803796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.804785 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.804789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:02:09.806557 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:02:09.806568 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.807673 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.807679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:02:09.807873 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:09.807913 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 19:02:09.808026 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 19:02:09.808159 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":1}
I20260924 19:02:09.808506 12217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:09.803006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41625 (local address 127.0.97.1:38365)
0924 19:02:09.803183 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 19:02:09.803191 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:02:09.803214 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:02:09.803306 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:02:09.803310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:02:09.803364 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:02:09.803465 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:02:09.803930 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.804629 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.806700 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.807523 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.807663 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:09.807759 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 19:02:09.808159 (+ 400us) server_negotiation.cc:299] Negotiation successful
0924 19:02:09.808294 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:02:09.809387 11877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea5897497344d15b64488945a46ce9f" candidate_uuid: "b47cba426e604b94811d61195672a6a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e5286592eb5486cb8b93d1bdc6d98da" is_pre_election: true
I20260924 19:02:09.809950 11877 raft_consensus.cc:2454] T dea5897497344d15b64488945a46ce9f P 0e5286592eb5486cb8b93d1bdc6d98da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b47cba426e604b94811d61195672a6a1 in term 0.
I20260924 19:02:09.810151 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:09.802868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60251 (local address 127.0.97.2:44285)
0924 19:02:09.803094 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 19:02:09.803102 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:02:09.804167 (+ 1065us) server_negotiation.cc:407] Connection header received
0924 19:02:09.804343 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:02:09.804349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:02:09.804433 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:02:09.804531 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:02:09.805108 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.806115 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.808128 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.809199 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.809337 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:09.809426 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 19:02:09.809757 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 19:02:09.809901 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 19:02:09.810267 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:09.802848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44285 (local address 127.0.97.3:60251)
0924 19:02:09.804065 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0924 19:02:09.804079 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:02:09.804213 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:02:09.804546 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:02:09.804550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:02:09.804572 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:02:09.804945 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:02:09.804954 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.806249 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.806254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:02:09.807936 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:02:09.807948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.809381 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.809387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:02:09.809607 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:09.809628 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 19:02:09.809726 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 19:02:09.809890 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1112,"mutex_wait_us":44,"thread_start_us":904,"threads_started":1}
I20260924 19:02:09.810671 12090 leader_election.cc:302] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e5286592eb5486cb8b93d1bdc6d98da, b47cba426e604b94811d61195672a6a1; no voters:
I20260924 19:02:09.811090 12224 raft_consensus.cc:2781] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:02:09.811247 12224 raft_consensus.cc:482] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:02:09.811359 12224 raft_consensus.cc:3037] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:02:09.811396 12020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea5897497344d15b64488945a46ce9f" candidate_uuid: "b47cba426e604b94811d61195672a6a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "880376e464394ea39516002498a4fe51" is_pre_election: true
I20260924 19:02:09.811942 12020 raft_consensus.cc:2454] T dea5897497344d15b64488945a46ce9f P 880376e464394ea39516002498a4fe51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b47cba426e604b94811d61195672a6a1 in term 0.
I20260924 19:02:09.814416 12224 raft_consensus.cc:504] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b47cba426e604b94811d61195672a6a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37629 } } peers { permanent_uuid: "880376e464394ea39516002498a4fe51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44285 } } peers { permanent_uuid: "0e5286592eb5486cb8b93d1bdc6d98da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } }
I20260924 19:02:09.815004 12224 leader_election.cc:288] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1 [CANDIDATE]: Term 1 election: Requested vote from peers 880376e464394ea39516002498a4fe51 (127.0.97.2:44285), 0e5286592eb5486cb8b93d1bdc6d98da (127.0.97.1:38365)
I20260924 19:02:09.815371 12020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea5897497344d15b64488945a46ce9f" candidate_uuid: "b47cba426e604b94811d61195672a6a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "880376e464394ea39516002498a4fe51"
I20260924 19:02:09.815570 12020 raft_consensus.cc:3037] T dea5897497344d15b64488945a46ce9f P 880376e464394ea39516002498a4fe51 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:02:09.815543 11877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea5897497344d15b64488945a46ce9f" candidate_uuid: "b47cba426e604b94811d61195672a6a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e5286592eb5486cb8b93d1bdc6d98da"
I20260924 19:02:09.815748 11877 raft_consensus.cc:3037] T dea5897497344d15b64488945a46ce9f P 0e5286592eb5486cb8b93d1bdc6d98da [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:02:09.818097 12020 raft_consensus.cc:2454] T dea5897497344d15b64488945a46ce9f P 880376e464394ea39516002498a4fe51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b47cba426e604b94811d61195672a6a1 in term 1.
I20260924 19:02:09.818163 11877 raft_consensus.cc:2454] T dea5897497344d15b64488945a46ce9f P 0e5286592eb5486cb8b93d1bdc6d98da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b47cba426e604b94811d61195672a6a1 in term 1.
I20260924 19:02:09.818540 12088 leader_election.cc:302] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 880376e464394ea39516002498a4fe51, b47cba426e604b94811d61195672a6a1; no voters:
I20260924 19:02:09.818859 12224 raft_consensus.cc:2781] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:02:09.819290 12224 raft_consensus.cc:686] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1 [term 1 LEADER]: Becoming Leader. State: Replica: b47cba426e604b94811d61195672a6a1, State: Running, Role: LEADER
I20260924 19:02:09.819736 12224 consensus_queue.cc:237] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b47cba426e604b94811d61195672a6a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37629 } } peers { permanent_uuid: "880376e464394ea39516002498a4fe51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44285 } } peers { permanent_uuid: "0e5286592eb5486cb8b93d1bdc6d98da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } }
I20260924 19:02:09.824393 11708 catalog_manager.cc:5495] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to b47cba426e604b94811d61195672a6a1 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "b47cba426e604b94811d61195672a6a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b47cba426e604b94811d61195672a6a1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "880376e464394ea39516002498a4fe51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e5286592eb5486cb8b93d1bdc6d98da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 } health_report { overall_health: UNKNOWN } } }
I20260924 19:02:09.862511 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:09.856902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53946 (local address 127.0.97.3:37629)
0924 19:02:09.857118 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 19:02:09.857124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:02:09.857143 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:02:09.857272 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:02:09.857282 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:02:09.857363 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:02:09.857531 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:02:09.857891 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.858576 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.860394 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.860779 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.860914 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:09.861432 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:02:09.861937 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:02:09.862089 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 19:02:09.862191 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 19:02:09.862263 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 19:02:09.870110 12020 raft_consensus.cc:1261] T dea5897497344d15b64488945a46ce9f P 880376e464394ea39516002498a4fe51 [term 1 FOLLOWER]: Refusing update from remote peer b47cba426e604b94811d61195672a6a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:02:09.870123 11877 raft_consensus.cc:1261] T dea5897497344d15b64488945a46ce9f P 0e5286592eb5486cb8b93d1bdc6d98da [term 1 FOLLOWER]: Refusing update from remote peer b47cba426e604b94811d61195672a6a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:02:09.871018 12228 consensus_queue.cc:1035] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "880376e464394ea39516002498a4fe51" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:02:09.871389 12224 consensus_queue.cc:1035] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e5286592eb5486cb8b93d1bdc6d98da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:02:09.883731 12232 mvcc.cc:205] Tried to move back new op lower bound from 7332972666338660352 to 7332972666145521664. Current Snapshot: MvccSnapshot[applied={T|T < 7332972666338660352}]
I20260924 19:02:09.887931 12234 mvcc.cc:205] Tried to move back new op lower bound from 7332972666338660352 to 7332972666145521664. Current Snapshot: MvccSnapshot[applied={T|T < 7332972666338660352}]
I20260924 19:02:09.896767 11708 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:02:09.897047 11708 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:02:09.900244 11708 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3f7ef40f8e74175bdbd6213bc5b145c: Sending DeleteTablet for 3 replicas of tablet dea5897497344d15b64488945a46ce9f
I20260924 19:02:09.901319 12134 tablet_service.cc:1514] Processing DeleteTablet for tablet dea5897497344d15b64488945a46ce9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:02:09 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53944
I20260924 19:02:09.901727 11857 tablet_service.cc:1514] Processing DeleteTablet for tablet dea5897497344d15b64488945a46ce9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:02:09 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41730
I20260924 19:02:09.902195 12000 tablet_service.cc:1514] Processing DeleteTablet for tablet dea5897497344d15b64488945a46ce9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:02:09 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37806
I20260924 19:02:09.904564 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 11789
I20260924 19:02:09.918366 12242 tablet_replica.cc:330] T dea5897497344d15b64488945a46ce9f P 0e5286592eb5486cb8b93d1bdc6d98da: stopping tablet replica
I20260924 19:02:09.918962 12243 tablet_replica.cc:330] T dea5897497344d15b64488945a46ce9f P 880376e464394ea39516002498a4fe51: stopping tablet replica
I20260924 19:02:09.919075 12241 tablet_replica.cc:330] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1: stopping tablet replica
I20260924 19:02:09.919505 12241 raft_consensus.cc:2229] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:02:09.919505 12243 raft_consensus.cc:2229] T dea5897497344d15b64488945a46ce9f P 880376e464394ea39516002498a4fe51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:02:09.920009 12243 raft_consensus.cc:2258] T dea5897497344d15b64488945a46ce9f P 880376e464394ea39516002498a4fe51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:02:09.920106 12241 raft_consensus.cc:2258] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:02:09.920773 12242 raft_consensus.cc:2229] T dea5897497344d15b64488945a46ce9f P 0e5286592eb5486cb8b93d1bdc6d98da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:02:09.921182 12242 raft_consensus.cc:2258] T dea5897497344d15b64488945a46ce9f P 0e5286592eb5486cb8b93d1bdc6d98da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:02:09.924430 12241 ts_tablet_manager.cc:1905] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 19:02:09 dist-test-slave-9fn1 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 19:02:09.931327 12243 ts_tablet_manager.cc:1905] T dea5897497344d15b64488945a46ce9f P 880376e464394ea39516002498a4fe51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:02:09.931602 12241 ts_tablet_manager.cc:1918] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:02:09.931802 12241 log.cc:1196] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-2/wal/wals/dea5897497344d15b64488945a46ce9f
I20260924 19:02:09.932396 12241 ts_tablet_manager.cc:1939] T dea5897497344d15b64488945a46ce9f P b47cba426e604b94811d61195672a6a1: Deleting consensus metadata
I20260924 19:02:09.933094 12242 ts_tablet_manager.cc:1905] T dea5897497344d15b64488945a46ce9f P 0e5286592eb5486cb8b93d1bdc6d98da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:02:09.934129 11696 catalog_manager.cc:4841] TS b47cba426e604b94811d61195672a6a1 (127.0.97.3:37629): tablet dea5897497344d15b64488945a46ce9f (table test-table [id=06bb2af41dd94aaea94464bfc1bd1d39]) successfully deleted
I20260924 19:02:09.934732 12217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:09.906077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41746 (local address 127.0.97.1:38365)
0924 19:02:09.906310 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 19:02:09.906318 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:02:09.916606 (+ 10288us) server_negotiation.cc:407] Connection header received
0924 19:02:09.916870 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:02:09.916880 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:02:09.917024 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:02:09.917181 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:02:09.919537 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.920522 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.922050 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:09.922503 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:09.922673 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:09.927280 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:02:09.927294 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:02:09.927310 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:02:09.927346 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:02:09.931464 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:09.932317 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:09.932326 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:09.932330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:09.932488 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:09.932884 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:09.932888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:09.932891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:09.933281 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:02:09.933545 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0924 19:02:09.934106 (+ 561us) server_negotiation.cc:299] Negotiation successful
0924 19:02:09.934322 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 19:02:09.936539 11917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:02:09.941795 12243 ts_tablet_manager.cc:1918] T dea5897497344d15b64488945a46ce9f P 880376e464394ea39516002498a4fe51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:02:09.942034 12243 log.cc:1196] T dea5897497344d15b64488945a46ce9f P 880376e464394ea39516002498a4fe51: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276386554690-388-0/minicluster-data/ts-1/wal/wals/dea5897497344d15b64488945a46ce9f
I20260924 19:02:09.942495 12243 ts_tablet_manager.cc:1939] T dea5897497344d15b64488945a46ce9f P 880376e464394ea39516002498a4fe51: Deleting consensus metadata
I20260924 19:02:09.944305 11694 catalog_manager.cc:4841] TS 880376e464394ea39516002498a4fe51 (127.0.97.2:44285): tablet dea5897497344d15b64488945a46ce9f (table test-table [id=06bb2af41dd94aaea94464bfc1bd1d39]) successfully deleted
I20260924 19:02:10.027504 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 11789
W20260924 19:02:10.044920 11696 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:38365 (error 108)
I20260924 19:02:10.045477 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 11928
I20260924 19:02:10.046871 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:10.046410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38365 (local address 127.0.0.1:41762)
0924 19:02:10.046596 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 19:02:10.046695 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 19:02:10.047129 11696 catalog_manager.cc:4584] TS 0e5286592eb5486cb8b93d1bdc6d98da (127.0.97.1:38365): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dea5897497344d15b64488945a46ce9f: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38365: connect: Connection refused (error 111)
Sep 24 19:02:10 dist-test-slave-9fn1 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 19:02:10.062537 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:10.046796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37818 (local address 127.0.97.2:44285)
0924 19:02:10.046967 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 19:02:10.046973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:02:10.047177 (+ 204us) server_negotiation.cc:407] Connection header received
0924 19:02:10.047250 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:02:10.047256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:02:10.047348 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:02:10.047448 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:02:10.048732 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:10.050041 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:10.051278 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:10.051942 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:10.052075 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:10.056104 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:02:10.056123 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:02:10.056142 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:02:10.056199 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:02:10.059873 (+ 3674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:10.060481 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:10.060491 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:10.060495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:10.060648 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:10.060976 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:10.060981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:10.060985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:10.061421 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:02:10.061760 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0924 19:02:10.061925 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 19:02:10.062140 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 19:02:10.063239 12060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:02:10.100919 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:10.100387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38365 (local address 127.0.0.1:41778)
0924 19:02:10.100632 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 19:02:10.100749 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260924 19:02:10.102098 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:10.101701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38365 (local address 127.0.0.1:41788)
0924 19:02:10.101860 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 19:02:10.101960 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 19:02:10.155414 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 11928
I20260924 19:02:10.162591 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:10.162046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38365 (local address 127.0.0.1:41792)
0924 19:02:10.162285 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 19:02:10.162394 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 19:02:10.163836 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:10.163403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38365 (local address 127.0.0.1:41802)
0924 19:02:10.163584 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 19:02:10.163678 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 19:02:10.173357 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 12071
Sep 24 19:02:10 dist-test-slave-9fn1 krb5kdc[11233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 19:02:10.191576 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:10.174168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53952 (local address 127.0.97.3:37629)
0924 19:02:10.174335 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 19:02:10.174340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:02:10.174357 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:02:10.174555 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:02:10.174559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:02:10.174618 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:02:10.174707 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:02:10.176433 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:10.177463 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:10.179627 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:10.180120 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:10.180254 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:10.185480 (+ 5226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:02:10.185492 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:02:10.185659 (+ 167us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:02:10.185700 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:02:10.188694 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:10.189389 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:10.189396 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:10.189399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:10.189607 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:10.190015 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:10.190020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:10.190023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:10.190358 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:02:10.190693 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0924 19:02:10.190991 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 19:02:10.191204 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:02:10.192222 12194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:02:10.244424 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:10.243931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38365 (local address 127.0.0.1:41818)
0924 19:02:10.244124 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 19:02:10.244248 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 19:02:10.245406 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:10.245044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38365 (local address 127.0.0.1:41832)
0924 19:02:10.245237 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 19:02:10.245307 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 19:02:10.314141 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 12071
I20260924 19:02:10.336529 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 11677
I20260924 19:02:10.338030 11739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:02:10.449648 12216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:10.448409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38365 (local address 127.0.0.1:41846)
0924 19:02:10.449238 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0924 19:02:10.449366 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":701}
I20260924 19:02:10.449875 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 11677
I20260924 19:02:10.469925 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 19:02:10.805553 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 19:02:10.805789 388 mini_ranger.cc:66] Stopping Ranger...
I20260924 19:02:11.150925 388 mini_ranger.cc:68] Stopped Ranger
2026-09-24 19:02:11.151 UTC [11263] LOG: received smart shutdown request
2026-09-24 19:02:11.154 UTC [11263] LOG: background worker "logical replication launcher" (PID 11272) exited with exit code 1
2026-09-24 19:02:11.154 UTC [11267] LOG: shutting down
2026-09-24 19:02:11.342 UTC [11263] LOG: database system is shut down
2026-09-24T19:02:11Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49147 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:02:11 dist-test-slave-9fn1 krb5kdc[12258](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:02:11 dist-test-slave-9fn1 krb5kdc[12258](info): set up 2 sockets
Sep 24 19:02:11 dist-test-slave-9fn1 krb5kdc[12258](info): commencing operation
krb5kdc: starting...
W20260924 19:02:12.573797 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:02:12 dist-test-slave-9fn1 krb5kdc[12258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:02:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:02:12Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260924 19:02:12.897149 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskLphABj/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/postgres -l logfile start
2026-09-24 19:02:15.506 UTC [12288] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 19:02:15.506 UTC [12288] LOG: listening on IPv4 address "127.0.97.20", port 33197
2026-09-24 19:02:15.509 UTC [12288] LOG: listening on Unix socket "/tmp/.s.PGSQL.33197"
2026-09-24 19:02:15.531 UTC [12291] LOG: database system was shut down at 2026-09-24 19:02:13 UTC
2026-09-24 19:02:15.534 UTC [12288] LOG: database system is ready to accept connections
I20260924 19:02:16.565611 388 mini_postgres.cc:96] Postgres bound to 33197
2026-09-24 19:02:16.589 UTC [12299] FATAL: database "slave" does not exist
127.0.97.20:33197 - accepting connections
I20260924 19:02:16.590350 388 mini_ranger.cc:161] Starting Ranger...
I20260924 19:02:16.616266 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 19:02:17.147513 388 mini_ranger.cc:87] Created ranger Postgres database
I20260924 19:02:17.147634 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ranger-admin
2026-09-24 19:02:17,420 [I] DB FLAVOR :POSTGRES
2026-09-24 19:02:17,421 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 19:02:17,421 [I] Checking connection..
2026-09-24 19:02:17,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 19:02:17,683 [I] Checking connection passed.
2026-09-24 19:02:17,684 [I] --------- Verifying version history table ---------
2026-09-24 19:02:17,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:02:17,924 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 19:02:17,925 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:02:18,173 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 19:02:18,173 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 19:02:18,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33197/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 19:02:18.421 UTC [12363] WARNING: there is no transaction in progress
2026-09-24 19:02:18,428 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 19:02:18,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:02:18,661 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 19:02:18,662 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 19:02:18,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 19:02:18,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 19:02:19,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-9fn1.c.gcp-upstream.internal','N') ;"
2026-09-24 19:02:19,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 19:02:19,643 [I] Table x_portal_user does not exist in database ranger
2026-09-24 19:02:19,644 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 19:02:19,877 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 19:02:19,878 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 19:02:20,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 19:02:20,336 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 19:02:20,337 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33197/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 19:02:21.302 UTC [12471] WARNING: there is no transaction in progress
2026-09-24 19:02:21.318 UTC [12471] WARNING: there is no transaction in progress
2026-09-24 19:02:21.335 UTC [12471] WARNING: there is no transaction in progress
2026-09-24 19:02:21.351 UTC [12471] WARNING: there is no transaction in progress
2026-09-24 19:02:21.368 UTC [12471] WARNING: there is no transaction in progress
2026-09-24 19:02:21.386 UTC [12471] WARNING: there is no transaction in progress
2026-09-24 19:02:21.442 UTC [12471] WARNING: there is no transaction in progress
2026-09-24 19:02:21.450 UTC [12471] WARNING: there is no transaction in progress
2026-09-24 19:02:21.464 UTC [12471] WARNING: there is no transaction in progress
2026-09-24 19:02:21.474 UTC [12471] WARNING: there is no transaction in progress
2026-09-24 19:02:21.485 UTC [12471] WARNING: there is no transaction in progress
2026-09-24 19:02:21.498 UTC [12471] WARNING: there is no transaction in progress
2026-09-24 19:02:21.508 UTC [12471] WARNING: there is no transaction in progress
2026-09-24 19:02:21.514 UTC [12471] WARNING: there is no transaction in progress
2026-09-24 19:02:21.521 UTC [12471] WARNING: there is no transaction in progress
2026-09-24 19:02:21.529 UTC [12471] WARNING: there is no transaction in progress
2026-09-24 19:02:22,185 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 19:02:22,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 19:02:22,423 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 19:02:22,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 19:02:22,668 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 19:02:22,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 19:02:22,920 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 19:02:22,921 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 19:02:23,164 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 19:02:23,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-9fn1.c.gcp-upstream.internal';"
2026-09-24 19:02:23,645 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 19:02:23,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 19:02:23,896 [I] DB_PATCHES have already been applied
I20260924 19:02:23.902563 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskLphABj/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 19:02:23.902640 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260924 19:02:23.916638 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:47857
Sep 24, 2026 7:02:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:02:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:02:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 7:02:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 7:02:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 7:02:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 7:02:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 19:02:25.018772 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.428s user 0.004s sys 0.032s
Sep 24, 2026 7:02:45 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 7:02:45 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 7:02:45 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 7:02:45 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 7:02:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 7:02:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 7:02:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 7:02:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 7:02:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 7:02:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 7:02:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 7:02:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 7:02:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 7:02:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 7:02:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 7:02:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 7:02:45 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 7:02:46 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 19:02:47.183714 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 19:02:47 dist-test-slave-9fn1 krb5kdc[12258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276567, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 19:02:47.360131 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 19:02:47.379889 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 19:02:48.229401 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 19:02:48.229552 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ranger-kms
I20260924 19:02:48.229612 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:33197
2026-09-24 19:02:48,574 [I] DB FLAVOR :POSTGRES
2026-09-24 19:02:48,574 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 19:02:48,574 [I] Checking connection
2026-09-24 19:02:48,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33197/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 19:02:48,801 [I] connection success
2026-09-24 19:02:48,802 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 19:02:48,802 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 19:02:48,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33197/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 19:02:49,026 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 19:02:49,026 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 19:02:49,026 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 19:02:49,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:33197/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 19:02:49,270 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 19:02:49.732234 388 mini_ranger_kms.cc:310] Created kms service
I20260924 19:02:49.912511 388 mini_ranger_kms.cc:326] Created kudu user
I20260924 19:02:49.955982 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 19:02:49 dist-test-slave-9fn1 krb5kdc[12258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276567, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 19:02:50.345654 388 mini_ranger_kms.cc:384] Added ranger policy
I20260924 19:02:50.345992 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskLphABj/build/asan/bin/postgresql.jar:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskLphABj/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 19:02:50.346057 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260924 19:02:50.364609 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12651
I20260924 19:02:50.364737 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:33501
Sep 24, 2026 7:02:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:02:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:02:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 7:02:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 7:02:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 7:02:51 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 24 19:02:52 dist-test-slave-9fn1 krb5kdc[12258](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790276572, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:02:52.670167 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.310s user 0.073s sys 0.280s
I20260924 19:02:52.670411 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 19:02:52.670585 388 mini_ranger_kms.cc:401] 127.0.97.20:33501/kms/v1/keys
Sep 24 19:02:53 dist-test-slave-9fn1 krb5kdc[12258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276572, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 24 19:02:53 dist-test-slave-9fn1 krb5kdc[12258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276572, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 24 19:02:55 dist-test-slave-9fn1 krb5kdc[12258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 19:02:55.557279 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44897
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44897
--ranger_config_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33501/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:02:55.867264 12702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:02:55.867600 12702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:02:55.867769 12702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:02:55.874730 12702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:02:55.874886 12702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:02:55.874997 12702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:02:55.875100 12702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:02:55.875175 12702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:02:55.884903 12702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:33501/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44897
--ranger_config_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:02:55.886171 12702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:02:55.887446 12702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:02:55.899294 12707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:02:55.899333 12708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:02:55.899257 12710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:02:55.899727 12702 server_base.cc:1034] running on GCE node
I20260924 19:02:55.900458 12702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:02:55.901892 12702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:02:55.903082 12702 hybrid_clock.cc:648] HybridClock initialized: now 1790276575903039 us; error 46 us; skew 500 ppm
I20260924 19:02:55.903409 12702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:02:55 dist-test-slave-9fn1 krb5kdc[12258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:02:55.910193 12702 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:02:55.911615 12702 webserver.cc:460] Webserver started at http://127.0.97.62:38133/ using document root <none> and password file <none>
I20260924 19:02:55.912271 12702 fs_manager.cc:359] Metadata directory not provided
I20260924 19:02:55.912395 12702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:02:55.912710 12702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:02:55 dist-test-slave-9fn1 krb5kdc[12258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 19:02:55.959208 12702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "6f7ea2d1a0e6429a87b1160d8f3d6746"
format_stamp: "Formatted at 2026-09-24 19:02:55 on dist-test-slave-9fn1"
server_key: "23499a9eb4efbf323de6326d62f589e2"
server_key_iv: "2cc68451eb36cc1536c59dd5870e293a"
server_key_version: "kuduclusterkey@0"
I20260924 19:02:55.960043 12702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "6f7ea2d1a0e6429a87b1160d8f3d6746"
format_stamp: "Formatted at 2026-09-24 19:02:55 on dist-test-slave-9fn1"
server_key: "23499a9eb4efbf323de6326d62f589e2"
server_key_iv: "2cc68451eb36cc1536c59dd5870e293a"
server_key_version: "kuduclusterkey@0"
I20260924 19:02:55.965237 12702 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:02:55.981052 12717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:02:55.983667 12702 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 19:02:55.983883 12702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "6f7ea2d1a0e6429a87b1160d8f3d6746"
format_stamp: "Formatted at 2026-09-24 19:02:55 on dist-test-slave-9fn1"
server_key: "23499a9eb4efbf323de6326d62f589e2"
server_key_iv: "2cc68451eb36cc1536c59dd5870e293a"
server_key_version: "kuduclusterkey@0"
I20260924 19:02:55.984114 12702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:02:56.003840 12702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:02:56.007772 12702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:02:56.008100 12702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:02:56.025287 12702 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44897
I20260924 19:02:56.025307 12768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44897 every 8 connection(s)
I20260924 19:02:56.026722 12702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:02:56.028090 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 12702
I20260924 19:02:56.028298 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:02:56.029122 388 external_mini_cluster.cc:1428] Setting key 0963b0b49ec5951817cc184748dfa3c8
I20260924 19:02:56.031486 12769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 19:02:56 dist-test-slave-9fn1 krb5kdc[12258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:02:56.044487 12769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746: Bootstrap starting.
I20260924 19:02:56.048630 12769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746: Neither blocks nor log segments found. Creating new log.
I20260924 19:02:56.049747 12769 log.cc:824] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746: Log is configured to *not* fsync() on all Append() calls
I20260924 19:02:56.052140 12772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:56.031192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46052 (local address 127.0.97.62:44897)
0924 19:02:56.031976 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0924 19:02:56.031987 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:02:56.032027 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:02:56.033630 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:02:56.033653 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:02:56.034051 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:02:56.034384 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:02:56.035963 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:56.037293 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:56.039175 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:56.039840 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:56.039973 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:56.044400 (+ 4427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:02:56.044410 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:02:56.044430 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:02:56.044485 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:02:56.048148 (+ 3663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:56.048813 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:56.048822 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:56.048832 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:56.049002 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:56.049379 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:56.049385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:56.049389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:56.050014 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:02:56.050331 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0924 19:02:56.050776 (+ 445us) server_negotiation.cc:299] Negotiation successful
0924 19:02:56.051187 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":382,"threads_started":1}
I20260924 19:02:56.052547 12769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746: No bootstrap required, opened a new log
I20260924 19:02:56.060169 12769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7ea2d1a0e6429a87b1160d8f3d6746" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44897 } }
I20260924 19:02:56.060595 12769 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:02:56.060705 12769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f7ea2d1a0e6429a87b1160d8f3d6746, State: Initialized, Role: FOLLOWER
I20260924 19:02:56.061241 12769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7ea2d1a0e6429a87b1160d8f3d6746" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44897 } }
I20260924 19:02:56.061479 12769 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:02:56.061687 12769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:02:56.061859 12769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:02:56.063939 12769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7ea2d1a0e6429a87b1160d8f3d6746" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44897 } }
I20260924 19:02:56.064567 12769 leader_election.cc:302] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f7ea2d1a0e6429a87b1160d8f3d6746; no voters:
I20260924 19:02:56.065109 12769 leader_election.cc:288] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:02:56.065239 12774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:02:56.066015 12774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [term 1 LEADER]: Becoming Leader. State: Replica: 6f7ea2d1a0e6429a87b1160d8f3d6746, State: Running, Role: LEADER
I20260924 19:02:56.066594 12774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7ea2d1a0e6429a87b1160d8f3d6746" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44897 } }
I20260924 19:02:56.067173 12769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:02:56.069242 12775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f7ea2d1a0e6429a87b1160d8f3d6746" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7ea2d1a0e6429a87b1160d8f3d6746" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44897 } } }
I20260924 19:02:56.069422 12776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f7ea2d1a0e6429a87b1160d8f3d6746. Latest consensus state: current_term: 1 leader_uuid: "6f7ea2d1a0e6429a87b1160d8f3d6746" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7ea2d1a0e6429a87b1160d8f3d6746" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44897 } } }
I20260924 19:02:56.069677 12775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [sys.catalog]: This master's current role is: LEADER
I20260924 19:02:56.069723 12776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [sys.catalog]: This master's current role is: LEADER
I20260924 19:02:56.071903 12782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:02:56.077394 12782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:02:56.078050 12769 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 19:02:56.100375 12782 catalog_manager.cc:1347] Generated new cluster ID: f19ccbe9ecb742469bf8c3720a5b59d3
I20260924 19:02:56.100585 12782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:02:56.134618 12782 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:02:56.136538 12782 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:02:56.145107 12782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746: Generated new TSK 0
I20260924 19:02:56.146311 12782 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 19:02:56.845082 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33501/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44897
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:02:57 dist-test-slave-9fn1 krb5kdc[12258](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790276577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 19:02:57.297147 12814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:02:57.297567 12814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:02:57.297771 12814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:02:57.309289 12814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:02:57.309525 12814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:02:57.309796 12814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:02:57.325383 12814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:33501/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:02:57.327466 12814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:02:57.329219 12814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:02:57.346931 12824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:02:57.355300 12828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:02:57.357465 12814 server_base.cc:1034] running on GCE node
W20260924 19:02:57.357769 12825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:02:57.358949 12814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:02:57.360266 12814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:02:57.361569 12814 hybrid_clock.cc:648] HybridClock initialized: now 1790276577361458 us; error 65 us; skew 500 ppm
I20260924 19:02:57.361956 12814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:02:57 dist-test-slave-9fn1 krb5kdc[12258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:02:57.365095 12814 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 19:02:57.370769 12814 webserver.cc:460] Webserver started at http://127.0.97.1:46331/ using document root <none> and password file <none>
I20260924 19:02:57.371572 12814 fs_manager.cc:359] Metadata directory not provided
I20260924 19:02:57.371706 12814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:02:57.372128 12814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:02:57 dist-test-slave-9fn1 krb5kdc[12258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 19:02:57.389761 12814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/data/instance:
uuid: "b52c4bfaf13b42d9b5feacb55cdc2780"
format_stamp: "Formatted at 2026-09-24 19:02:57 on dist-test-slave-9fn1"
server_key: "44b0722731fe4a742ea64b5a88f9cc34"
server_key_iv: "7cc6de91cf970207673bdf90fbc76525"
server_key_version: "kuduclusterkey@0"
I20260924 19:02:57.391674 12814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b52c4bfaf13b42d9b5feacb55cdc2780"
format_stamp: "Formatted at 2026-09-24 19:02:57 on dist-test-slave-9fn1"
server_key: "44b0722731fe4a742ea64b5a88f9cc34"
server_key_iv: "7cc6de91cf970207673bdf90fbc76525"
server_key_version: "kuduclusterkey@0"
I20260924 19:02:57.404714 12814 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.007s sys 0.004s
I20260924 19:02:57.434131 12834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:02:57.446842 12814 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.007s sys 0.004s
I20260924 19:02:57.447170 12814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "b52c4bfaf13b42d9b5feacb55cdc2780"
format_stamp: "Formatted at 2026-09-24 19:02:57 on dist-test-slave-9fn1"
server_key: "44b0722731fe4a742ea64b5a88f9cc34"
server_key_iv: "7cc6de91cf970207673bdf90fbc76525"
server_key_version: "kuduclusterkey@0"
I20260924 19:02:57.447536 12814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:02:57.496110 12814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:02:57.501600 12814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:02:57.501955 12814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:02:57.503105 12814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:02:57.504873 12814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:02:57.505017 12814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 19:02:57.505198 12814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:02:57.505329 12814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:02:57.575649 12814 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45819
I20260924 19:02:57.575654 12946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45819 every 8 connection(s)
I20260924 19:02:57.578116 12814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:02:57.587699 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 12814
I20260924 19:02:57.587898 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/wal/instance
I20260924 19:02:57.588685 388 external_mini_cluster.cc:1428] Setting key 6e9a580d1bd4605e048c6170a2d3e61e
Sep 24 19:02:57 dist-test-slave-9fn1 krb5kdc[12258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:02:57.621200 12950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:57.583730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41223 (local address 127.0.97.62:44897)
0924 19:02:57.593278 (+ 9548us) server_negotiation.cc:206] Beginning negotiation
0924 19:02:57.593286 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:02:57.593305 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:02:57.593357 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:02:57.593364 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:02:57.593488 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:02:57.593613 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:02:57.601182 (+ 7569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:57.602420 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:57.604869 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:57.605568 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:57.605665 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:57.613794 (+ 8129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:02:57.613801 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:02:57.613806 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:02:57.613847 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:02:57.617316 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:57.618513 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:57.618518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:57.618522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:57.618653 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:57.619207 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:57.619212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:57.619215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:57.619550 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:02:57.619659 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 19:02:57.620605 (+ 946us) server_negotiation.cc:299] Negotiation successful
0924 19:02:57.620815 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9407,"thread_start_us":17423,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 19:02:57.623728 12949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:57.583110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44897 (local address 127.0.97.1:41223)
0924 19:02:57.590063 (+ 6953us) negotiation.cc:107] Waiting for socket to connect
0924 19:02:57.590114 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 19:02:57.592559 (+ 2445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:02:57.599397 (+ 6838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:02:57.599419 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:02:57.600089 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:02:57.600941 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:02:57.600970 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:02:57.602635 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:02:57.602640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:02:57.604694 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:02:57.604717 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:02:57.606045 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:02:57.606053 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:02:57.606348 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:57.609734 (+ 3386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:02:57.609763 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:02:57.613245 (+ 3482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:02:57.617792 (+ 4547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:02:57.617803 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:02:57.617821 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:02:57.618382 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:02:57.618785 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:02:57.618789 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:02:57.618811 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:02:57.619085 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:02:57.619772 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:02:57.619792 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:02:57.620190 (+ 398us) client_negotiation.cc:769] Sending connection context
0924 19:02:57.621847 (+ 1657us) client_negotiation.cc:240] Negotiation successful
0924 19:02:57.622401 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6616,"thread_start_us":14259,"threads_started":1}
I20260924 19:02:57.625859 12947 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44897
I20260924 19:02:57.626212 12947 heartbeater.cc:464] Registering TS with master...
I20260924 19:02:57.627096 12947 heartbeater.cc:511] Master 127.0.97.62:44897 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260924 19:02:57.629623 12734 ts_manager.cc:194] Registered new tserver with Master: b52c4bfaf13b42d9b5feacb55cdc2780 (127.0.97.1:45819)
I20260924 19:02:57.631549 12734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41223
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 19:02:57.665334 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33501/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44897
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:02:57 dist-test-slave-9fn1 krb5kdc[12258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 24 19:02:57 dist-test-slave-9fn1 krb5kdc[12258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260924 19:02:58.129782 12953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:02:58.130235 12953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:02:58.130429 12953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:02:58.143062 12953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:02:58.143306 12953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:02:58.143581 12953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:02:58.160506 12953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:33501/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:02:58.162336 12953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:02:58.164120 12953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:02:58.187096 12961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:02:58.189908 12960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:02:58.197786 12963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:02:58.204239 12953 server_base.cc:1034] running on GCE node
I20260924 19:02:58.205360 12953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:02:58.207031 12953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:02:58.208729 12953 hybrid_clock.cc:648] HybridClock initialized: now 1790276578208634 us; error 55 us; skew 500 ppm
I20260924 19:02:58.209311 12953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:02:58 dist-test-slave-9fn1 krb5kdc[12258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:02:58.215987 12953 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 19:02:58.224681 12953 webserver.cc:460] Webserver started at http://127.0.97.2:33951/ using document root <none> and password file <none>
I20260924 19:02:58.225822 12953 fs_manager.cc:359] Metadata directory not provided
I20260924 19:02:58.226048 12953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:02:58.226459 12953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:02:58 dist-test-slave-9fn1 krb5kdc[12258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 19:02:58.248418 12953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/data/instance:
uuid: "12d14cd4fa0d4bbf9f1eee26a266fbd7"
format_stamp: "Formatted at 2026-09-24 19:02:58 on dist-test-slave-9fn1"
server_key: "3cebd36e728425e5a064817186bb4ff4"
server_key_iv: "93649ea444f67e311f95c4ab2077218f"
server_key_version: "kuduclusterkey@0"
I20260924 19:02:58.249521 12953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/wal/instance:
uuid: "12d14cd4fa0d4bbf9f1eee26a266fbd7"
format_stamp: "Formatted at 2026-09-24 19:02:58 on dist-test-slave-9fn1"
server_key: "3cebd36e728425e5a064817186bb4ff4"
server_key_iv: "93649ea444f67e311f95c4ab2077218f"
server_key_version: "kuduclusterkey@0"
I20260924 19:02:58.267639 12953 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.009s sys 0.007s
I20260924 19:02:58.339165 12970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:02:58.342662 12953 fs_manager.cc:725] Time spent opening block manager: real 0.027s user 0.006s sys 0.020s
I20260924 19:02:58.342975 12953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "12d14cd4fa0d4bbf9f1eee26a266fbd7"
format_stamp: "Formatted at 2026-09-24 19:02:58 on dist-test-slave-9fn1"
server_key: "3cebd36e728425e5a064817186bb4ff4"
server_key_iv: "93649ea444f67e311f95c4ab2077218f"
server_key_version: "kuduclusterkey@0"
I20260924 19:02:58.343356 12953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:02:58.369437 12953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:02:58.373574 12953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:02:58.373898 12953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:02:58.374912 12953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:02:58.376348 12953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:02:58.376447 12953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:02:58.376600 12953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:02:58.376672 12953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:02:58.425802 12953 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42719
I20260924 19:02:58.425855 13089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42719 every 8 connection(s)
I20260924 19:02:58.427409 12953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:02:58.437292 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 12953
I20260924 19:02:58.437469 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/wal/instance
I20260924 19:02:58.438156 388 external_mini_cluster.cc:1428] Setting key 16c1f94458ae0fcf8a4eab5bac9165de
Sep 24 19:02:58 dist-test-slave-9fn1 krb5kdc[12258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:02:58.466416 13092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:58.430993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44897 (local address 127.0.97.2:50145)
0924 19:02:58.438112 (+ 7119us) negotiation.cc:107] Waiting for socket to connect
0924 19:02:58.438163 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 19:02:58.440629 (+ 2466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:02:58.445888 (+ 5259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:02:58.445901 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:02:58.446558 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:02:58.447393 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:02:58.447420 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:02:58.449025 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:02:58.449030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:02:58.451197 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:02:58.451222 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:02:58.452632 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:02:58.452641 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:02:58.453015 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:58.453969 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:02:58.454002 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:02:58.456742 (+ 2740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:02:58.461114 (+ 4372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:02:58.461125 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:02:58.461147 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:02:58.461868 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:02:58.462360 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:02:58.462365 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:02:58.462369 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:02:58.462664 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:02:58.463456 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:02:58.463467 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:02:58.463955 (+ 488us) client_negotiation.cc:769] Sending connection context
0924 19:02:58.464302 (+ 347us) client_negotiation.cc:240] Negotiation successful
0924 19:02:58.464885 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6785,"thread_start_us":17743,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 19:02:58.467664 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:58.431594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50145 (local address 127.0.97.62:44897)
0924 19:02:58.441413 (+ 9819us) server_negotiation.cc:206] Beginning negotiation
0924 19:02:58.441421 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:02:58.441458 (+ 37us) server_negotiation.cc:407] Connection header received
0924 19:02:58.441532 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:02:58.441540 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:02:58.441630 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:02:58.441739 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:02:58.447626 (+ 5887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:58.448805 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:58.451366 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:58.452183 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:58.452354 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:58.456982 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:02:58.456992 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:02:58.456999 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:02:58.457045 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:02:58.460910 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:58.462023 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:58.462031 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:58.462038 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:58.462199 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:58.462805 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:58.462811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:58.462817 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:58.463202 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:02:58.463376 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 19:02:58.466940 (+ 3564us) server_negotiation.cc:299] Negotiation successful
0924 19:02:58.467266 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9675,"thread_start_us":13926,"threads_started":1}
I20260924 19:02:58.469650 13090 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44897
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260924 19:02:58.469981 13090 heartbeater.cc:464] Registering TS with master...
I20260924 19:02:58.470881 13090 heartbeater.cc:511] Master 127.0.97.62:44897 requested a full tablet report, sending...
I20260924 19:02:58.477265 12734 ts_manager.cc:194] Registered new tserver with Master: 12d14cd4fa0d4bbf9f1eee26a266fbd7 (127.0.97.2:42719)
I20260924 19:02:58.478170 12734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50145
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 19:02:58.500648 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33501/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44897
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:02:58.635059 12947 heartbeater.cc:503] Master 127.0.97.62:44897 was elected leader, sending a full tablet report...
W20260924 19:02:58.836822 13096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:02:58.837098 13096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:02:58.837248 13096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:02:58.844321 13096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:02:58.844425 13096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:02:58.844627 13096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:02:58.854722 13096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:33501/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:02:58.855968 13096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:02:58.857187 13096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:02:58.869887 13102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:02:58.869858 13104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:02:58.869932 13101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:02:58.870744 13096 server_base.cc:1034] running on GCE node
I20260924 19:02:58.871377 13096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:02:58.872349 13096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:02:58.873594 13096 hybrid_clock.cc:648] HybridClock initialized: now 1790276578873560 us; error 47 us; skew 500 ppm
I20260924 19:02:58.873899 13096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:02:58 dist-test-slave-9fn1 krb5kdc[12258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:02:58.877305 13096 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 19:02:58.878785 13096 webserver.cc:460] Webserver started at http://127.0.97.3:32859/ using document root <none> and password file <none>
I20260924 19:02:58.879429 13096 fs_manager.cc:359] Metadata directory not provided
I20260924 19:02:58.879570 13096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:02:58.879865 13096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:02:58 dist-test-slave-9fn1 krb5kdc[12258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 19:02:58.896371 13096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/data/instance:
uuid: "b86ccf4c1df94b9fb5af66ccadcda0d0"
format_stamp: "Formatted at 2026-09-24 19:02:58 on dist-test-slave-9fn1"
server_key: "9508b06c89dca3303e6e775bdd1e049d"
server_key_iv: "ca8052dbffad51f1df4c2cedd353d600"
server_key_version: "kuduclusterkey@0"
I20260924 19:02:58.897178 13096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b86ccf4c1df94b9fb5af66ccadcda0d0"
format_stamp: "Formatted at 2026-09-24 19:02:58 on dist-test-slave-9fn1"
server_key: "9508b06c89dca3303e6e775bdd1e049d"
server_key_iv: "ca8052dbffad51f1df4c2cedd353d600"
server_key_version: "kuduclusterkey@0"
I20260924 19:02:58.902618 13096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 19:02:58.917956 13111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:02:58.919493 13096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 19:02:58.919659 13096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "b86ccf4c1df94b9fb5af66ccadcda0d0"
format_stamp: "Formatted at 2026-09-24 19:02:58 on dist-test-slave-9fn1"
server_key: "9508b06c89dca3303e6e775bdd1e049d"
server_key_iv: "ca8052dbffad51f1df4c2cedd353d600"
server_key_version: "kuduclusterkey@0"
I20260924 19:02:58.919876 13096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:02:58.941329 13096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:02:58.945255 13096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:02:58.945600 13096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:02:58.946532 13096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:02:58.947817 13096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:02:58.947890 13096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:02:58.948047 13096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:02:58.948099 13096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:02:58.980583 13096 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37785
I20260924 19:02:58.980624 13223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37785 every 8 connection(s)
I20260924 19:02:58.981925 13096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:02:58.985308 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 13096
I20260924 19:02:58.985532 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/wal/instance
I20260924 19:02:58.986091 388 external_mini_cluster.cc:1428] Setting key bf229a46a3f6891a14445d71f7342eb7
Sep 24 19:02:58 dist-test-slave-9fn1 krb5kdc[12258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:02:59.001973 13226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:58.984363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55147 (local address 127.0.97.62:44897)
0924 19:02:58.984832 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0924 19:02:58.984839 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:02:58.986002 (+ 1163us) server_negotiation.cc:407] Connection header received
0924 19:02:58.988245 (+ 2243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:02:58.988250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:02:58.988355 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:02:58.988467 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:02:58.990116 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:58.990823 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:58.992324 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:58.992733 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:58.992861 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:58.996037 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:02:58.996042 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:02:58.996045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:02:58.996089 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:02:58.998613 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:58.999308 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:58.999312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:58.999314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:58.999429 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:59.000050 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:59.000053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:59.000055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:59.000419 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:02:59.000587 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 19:02:59.001335 (+ 748us) server_negotiation.cc:299] Negotiation successful
0924 19:02:59.001582 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":200,"threads_started":1}
I20260924 19:02:59.002919 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:58.984951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44897 (local address 127.0.97.3:55147)
0924 19:02:58.985836 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0924 19:02:58.985873 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:02:58.987976 (+ 2103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:02:58.988734 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:02:58.988747 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:02:58.989327 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:02:58.989958 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:02:58.989976 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:02:58.990971 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:02:58.990976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:02:58.992168 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:02:58.992186 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:02:58.992947 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:02:58.992954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:02:58.993187 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:58.993909 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:02:58.993929 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:02:58.995829 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:02:58.998755 (+ 2926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:02:58.998763 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:02:58.998778 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:02:58.999165 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:02:58.999548 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:02:58.999553 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:02:58.999557 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:02:58.999794 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:02:59.000556 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:02:59.000565 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:02:59.000974 (+ 409us) client_negotiation.cc:769] Sending connection context
0924 19:02:59.001239 (+ 265us) client_negotiation.cc:240] Negotiation successful
0924 19:02:59.001728 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":423,"threads_started":1}
I20260924 19:02:59.004101 13224 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44897
I20260924 19:02:59.004367 13224 heartbeater.cc:464] Registering TS with master...
I20260924 19:02:59.005026 13224 heartbeater.cc:511] Master 127.0.97.62:44897 requested a full tablet report, sending...
I20260924 19:02:59.006294 12734 ts_manager.cc:194] Registered new tserver with Master: b86ccf4c1df94b9fb5af66ccadcda0d0 (127.0.97.3:37785)
I20260924 19:02:59.007038 12734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:55147
I20260924 19:02:59.013109 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:02:59.014755 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 12814
Sep 24 19:02:59 dist-test-slave-9fn1 krb5kdc[12258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 19:02:59.031186 13229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:59.015702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56138 (local address 127.0.97.1:45819)
0924 19:02:59.016322 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0924 19:02:59.016328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:02:59.016350 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:02:59.016406 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:02:59.016412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:02:59.016586 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:02:59.016704 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:02:59.018061 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:59.019069 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:59.020522 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:59.021036 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:59.021140 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:59.025910 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:02:59.025921 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:02:59.025933 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:02:59.025964 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:02:59.028559 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:59.029214 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:59.029222 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:59.029225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:59.029390 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:59.029779 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:59.029785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:59.029788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:59.030085 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:02:59.030391 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0924 19:02:59.030613 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 19:02:59.030862 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":340,"threads_started":1}
I20260924 19:02:59.032299 12942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:02:59.114372 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 12814
I20260924 19:02:59.130153 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 12953
Sep 24 19:02:59 dist-test-slave-9fn1 krb5kdc[12258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 19:02:59.145403 13231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:59.131031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42464 (local address 127.0.97.2:42719)
0924 19:02:59.131529 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0924 19:02:59.131535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:02:59.131561 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:02:59.131620 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:02:59.131626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:02:59.131838 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:02:59.131971 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:02:59.133155 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:59.134196 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:59.135518 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:59.136032 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:59.136126 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:59.140331 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:02:59.140341 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:02:59.140355 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:02:59.140382 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:02:59.142963 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:59.143595 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:59.143601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:59.143603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:59.143725 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:59.144049 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:59.144053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:59.144055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:59.144358 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:02:59.144564 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 19:02:59.144798 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 19:02:59.145020 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":250,"threads_started":1}
I20260924 19:02:59.146600 13085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:02:59.227181 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 12953
I20260924 19:02:59.242664 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 13096
Sep 24 19:02:59 dist-test-slave-9fn1 krb5kdc[12258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 19:02:59.259013 13233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:02:59.243676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41020 (local address 127.0.97.3:37785)
0924 19:02:59.244290 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0924 19:02:59.244297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:02:59.244324 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:02:59.244379 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:02:59.244385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:02:59.244604 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:02:59.244779 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:02:59.246193 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:59.247179 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:59.248524 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:02:59.249070 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:02:59.249207 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:02:59.253610 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:02:59.253621 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:02:59.253635 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:02:59.253666 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:02:59.256630 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:59.257230 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:59.257238 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:59.257242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:59.257361 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:02:59.257718 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:02:59.257721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:02:59.257723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:02:59.258051 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:02:59.258316 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0924 19:02:59.258557 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 19:02:59.258731 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":297,"threads_started":1}
I20260924 19:02:59.260234 13219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:02:59.341784 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 13096
I20260924 19:02:59.357030 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 12702
I20260924 19:02:59.358222 12764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:02:59.462585 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 12702
I20260924 19:02:59.484882 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44897
--webserver_interface=127.0.97.62
--webserver_port=38133
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44897
--ranger_config_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33501/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:33501/kms with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:02:59.799101 13236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:02:59.799383 13236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:02:59.799535 13236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:02:59.806649 13236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:02:59.806774 13236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:02:59.806896 13236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:02:59.806993 13236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:02:59.807073 13236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:02:59.816774 13236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:33501/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44897
--ranger_config_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=38133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:02:59.818199 13236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:02:59.819486 13236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:02:59.831566 13241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:02:59.831523 13244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:02:59.831486 13242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:02:59.832235 13236 server_base.cc:1034] running on GCE node
I20260924 19:02:59.833065 13236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:02:59.834610 13236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:02:59.835866 13236 hybrid_clock.cc:648] HybridClock initialized: now 1790276579835811 us; error 38 us; skew 500 ppm
I20260924 19:02:59.836251 13236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:02:59 dist-test-slave-9fn1 krb5kdc[12258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:02:59.839347 13236 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:02:59.840835 13236 webserver.cc:460] Webserver started at http://127.0.97.62:38133/ using document root <none> and password file <none>
I20260924 19:02:59.841457 13236 fs_manager.cc:359] Metadata directory not provided
I20260924 19:02:59.841624 13236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 19:02:59 dist-test-slave-9fn1 krb5kdc[12258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 19:02:59.911414 13236 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 19:02:59.914348 13252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:02:59.915807 13236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:02:59.916010 13236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "6f7ea2d1a0e6429a87b1160d8f3d6746"
format_stamp: "Formatted at 2026-09-24 19:02:55 on dist-test-slave-9fn1"
server_key: "23499a9eb4efbf323de6326d62f589e2"
server_key_iv: "2cc68451eb36cc1536c59dd5870e293a"
server_key_version: "kuduclusterkey@0"
I20260924 19:02:59.916985 13236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:02:59.939601 13236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:02:59.943760 13236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:02:59.944067 13236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:02:59.962667 13236 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44897
I20260924 19:02:59.962687 13303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44897 every 8 connection(s)
I20260924 19:02:59.964174 13236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:02:59.966377 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 13236
I20260924 19:02:59.967386 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33501/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:45819
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:44897
--webserver_port=46331
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:33501/kms with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:02:59.973608 13304 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:02:59.977766 13304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746: Bootstrap starting.
I20260924 19:02:59.996631 13304 log.cc:824] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746: Log is configured to *not* fsync() on all Append() calls
I20260924 19:03:00.005128 13304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:03:00.005728 13304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746: Bootstrap complete.
I20260924 19:03:00.014433 13304 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7ea2d1a0e6429a87b1160d8f3d6746" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44897 } }
I20260924 19:03:00.015313 13304 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f7ea2d1a0e6429a87b1160d8f3d6746, State: Initialized, Role: FOLLOWER
I20260924 19:03:00.015945 13304 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7ea2d1a0e6429a87b1160d8f3d6746" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44897 } }
I20260924 19:03:00.016155 13304 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:03:00.016266 13304 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:03:00.016446 13304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:03:00.020319 13304 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7ea2d1a0e6429a87b1160d8f3d6746" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44897 } }
I20260924 19:03:00.020800 13304 leader_election.cc:302] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f7ea2d1a0e6429a87b1160d8f3d6746; no voters:
I20260924 19:03:00.021485 13304 leader_election.cc:288] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 19:03:00.021651 13308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:03:00.023881 13304 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:03:00.028350 13308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [term 2 LEADER]: Becoming Leader. State: Replica: 6f7ea2d1a0e6429a87b1160d8f3d6746, State: Running, Role: LEADER
I20260924 19:03:00.028931 13308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7ea2d1a0e6429a87b1160d8f3d6746" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44897 } }
I20260924 19:03:00.031756 13310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f7ea2d1a0e6429a87b1160d8f3d6746. Latest consensus state: current_term: 2 leader_uuid: "6f7ea2d1a0e6429a87b1160d8f3d6746" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7ea2d1a0e6429a87b1160d8f3d6746" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44897 } } }
I20260924 19:03:00.032182 13310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [sys.catalog]: This master's current role is: LEADER
I20260924 19:03:00.032961 13309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6f7ea2d1a0e6429a87b1160d8f3d6746" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7ea2d1a0e6429a87b1160d8f3d6746" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44897 } } }
I20260924 19:03:00.033304 13309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746 [sys.catalog]: This master's current role is: LEADER
I20260924 19:03:00.039705 13318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:03:00.046422 13318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:03:00.048282 13318 catalog_manager.cc:1259] Loaded cluster ID: f19ccbe9ecb742469bf8c3720a5b59d3
I20260924 19:03:00.048463 13318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:03:00.053618 13318 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:03:00.055507 13318 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746: Loaded TSK: 0
I20260924 19:03:00.057526 13318 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:03:00.060607 13304 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260924 19:03:00.425001 13306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:03:00.425477 13306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:03:00.425696 13306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:03:00.437582 13306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:03:00.437808 13306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:03:00.438041 13306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:03:00.454038 13306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:33501/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:45819
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=46331
--tserver_master_addrs=127.0.97.62:44897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:03:00.455828 13306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:03:00.457679 13306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:03:00.477705 13342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:00.485088 13341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:03:00.485965 13306 server_base.cc:1034] running on GCE node
W20260924 19:03:00.486501 13344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:03:00.487495 13306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:03:00.501235 13306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:03:00.502601 13306 hybrid_clock.cc:648] HybridClock initialized: now 1790276580502494 us; error 112 us; skew 500 ppm
I20260924 19:03:00.503187 13306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:03:00 dist-test-slave-9fn1 krb5kdc[12258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:03:00.510274 13306 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 19:03:00.529606 13306 webserver.cc:460] Webserver started at http://127.0.97.1:46331/ using document root <none> and password file <none>
I20260924 19:03:00.530750 13306 fs_manager.cc:359] Metadata directory not provided
I20260924 19:03:00.530977 13306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 19:03:00 dist-test-slave-9fn1 krb5kdc[12258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 19:03:00.596323 13306 fs_manager.cc:709] Time spent opening directory manager: real 0.034s user 0.020s sys 0.012s
I20260924 19:03:00.613838 13353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:00.616003 13306 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.003s sys 0.009s
I20260924 19:03:00.616328 13306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "b52c4bfaf13b42d9b5feacb55cdc2780"
format_stamp: "Formatted at 2026-09-24 19:02:57 on dist-test-slave-9fn1"
server_key: "44b0722731fe4a742ea64b5a88f9cc34"
server_key_iv: "7cc6de91cf970207673bdf90fbc76525"
server_key_version: "kuduclusterkey@0"
I20260924 19:03:00.618032 13306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:03:00.662297 13306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:03:00.667424 13306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:03:00.667783 13306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:03:00.668860 13306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:03:00.670521 13306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:03:00.670631 13306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:00.670723 13306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:03:00.670794 13306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:00.750641 13306 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45819
I20260924 19:03:00.750687 13465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45819 every 8 connection(s)
I20260924 19:03:00.752242 13306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:03:00.752939 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 13306
I20260924 19:03:00.753759 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33501/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42719
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:44897
--webserver_port=33951
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:33501/kms with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:03:00 dist-test-slave-9fn1 krb5kdc[12258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:03:00.796289 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:00.756309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44897 (local address 127.0.97.1:58289)
0924 19:03:00.763696 (+ 7387us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:00.763756 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:00.769902 (+ 6146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:00.772013 (+ 2111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:00.772028 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:00.772781 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:03:00.773686 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:00.773714 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:00.776362 (+ 2648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:00.776372 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:00.778977 (+ 2605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:00.778997 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:00.781229 (+ 2232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:00.781241 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:00.781659 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:00.782567 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:03:00.782618 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:03:00.785948 (+ 3330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:03:00.790952 (+ 5004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:00.790962 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:00.790980 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:00.791622 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:00.792162 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:00.792168 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:00.792172 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:00.792391 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:00.793466 (+ 1075us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:03:00.793475 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:03:00.793909 (+ 434us) client_negotiation.cc:769] Sending connection context
0924 19:03:00.794206 (+ 297us) client_negotiation.cc:240] Negotiation successful
0924 19:03:00.794786 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7053,"thread_start_us":6848,"threads_started":1}
I20260924 19:03:00.798723 13468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:00.756158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58289 (local address 127.0.97.62:44897)
0924 19:03:00.766012 (+ 9854us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:00.766034 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:00.766106 (+ 72us) server_negotiation.cc:407] Connection header received
0924 19:03:00.770787 (+ 4681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:00.770825 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:00.771366 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:00.771843 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:00.773923 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:00.776159 (+ 2236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:00.779203 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:00.780518 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:00.780764 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:00.786308 (+ 5544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:00.786323 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:00.786350 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:00.786420 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:00.790763 (+ 4343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:00.791799 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:00.791809 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:00.791818 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:00.792018 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:00.792521 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:00.792527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:00.792531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:00.793195 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:00.793385 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:00.797002 (+ 3617us) server_negotiation.cc:299] Negotiation successful
0924 19:03:00.797425 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9584,"thread_start_us":9355,"threads_started":1}
I20260924 19:03:00.803440 13466 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44897
I20260924 19:03:00.803803 13466 heartbeater.cc:464] Registering TS with master...
I20260924 19:03:00.804653 13466 heartbeater.cc:511] Master 127.0.97.62:44897 requested a full tablet report, sending...
W20260924 19:03:00.807821 13466 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:44897 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 19:03:00.808082 13466 heartbeater.cc:464] Registering TS with master...
I20260924 19:03:00.808444 13466 heartbeater.cc:511] Master 127.0.97.62:44897 requested a full tablet report, sending...
I20260924 19:03:00.809719 13466 heartbeater.cc:464] Registering TS with master...
I20260924 19:03:00.809993 13466 heartbeater.cc:511] Master 127.0.97.62:44897 requested a full tablet report, sending...
W20260924 19:03:00.811116 13466 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:03:01.179482 13470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:03:01.179811 13470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:03:01.179976 13470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:03:01.187803 13470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:03:01.187953 13470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:03:01.188169 13470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:03:01.202379 13470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:33501/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42719
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33951
--tserver_master_addrs=127.0.97.62:44897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:03:01.204218 13470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:03:01.205956 13470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:03:01.225189 13485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:01.225832 13484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:03:01.229776 13470 server_base.cc:1034] running on GCE node
W20260924 19:03:01.232802 13489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:03:01.233661 13470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:03:01.241138 13470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:03:01.242496 13470 hybrid_clock.cc:648] HybridClock initialized: now 1790276581242422 us; error 69 us; skew 500 ppm
I20260924 19:03:01.243049 13470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:03:01 dist-test-slave-9fn1 krb5kdc[12258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:03:01.255777 13470 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 19:03:01.257805 13470 webserver.cc:460] Webserver started at http://127.0.97.2:33951/ using document root <none> and password file <none>
I20260924 19:03:01.258574 13470 fs_manager.cc:359] Metadata directory not provided
I20260924 19:03:01.258741 13470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 19:03:01 dist-test-slave-9fn1 krb5kdc[12258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 19:03:01.281942 13470 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 19:03:01.284963 13497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:01.286510 13470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:03:01.286712 13470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "12d14cd4fa0d4bbf9f1eee26a266fbd7"
format_stamp: "Formatted at 2026-09-24 19:02:58 on dist-test-slave-9fn1"
server_key: "3cebd36e728425e5a064817186bb4ff4"
server_key_iv: "93649ea444f67e311f95c4ab2077218f"
server_key_version: "kuduclusterkey@0"
I20260924 19:03:01.287868 13470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:03:01.312047 13470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
Sep 24 19:03:01 dist-test-slave-9fn1 krb5kdc[12258](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790276581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:03:01.316792 13470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:03:01.317097 13470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:03:01.321894 13470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:03:01.338434 13470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:03:01.338632 13470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 19:03:01.338785 13470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:03:01.338910 13470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:01.385885 13470 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42719
I20260924 19:03:01.385934 13609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42719 every 8 connection(s)
I20260924 19:03:01.387364 13470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:03:01.387950 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 13470
I20260924 19:03:01.388891 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:33501/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:37785
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:44897
--webserver_port=32859
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:33501/kms with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:03:01 dist-test-slave-9fn1 krb5kdc[12258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:03:01.415720 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:01.391019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44897 (local address 127.0.97.2:47891)
0924 19:03:01.392426 (+ 1407us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:01.392486 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:01.395254 (+ 2768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:01.396070 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:01.396082 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:01.396677 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:03:01.397483 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:01.397545 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:01.399169 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:01.399175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:01.401177 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:01.401192 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:01.402541 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:01.402548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:01.402839 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:01.403671 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:03:01.403706 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:03:01.406449 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:03:01.410736 (+ 4287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:01.410745 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:01.410763 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:01.411378 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:01.411831 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:01.411836 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:01.411839 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:01.412092 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:01.412843 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:03:01.412851 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:03:01.413308 (+ 457us) client_negotiation.cc:769] Sending connection context
0924 19:03:01.413659 (+ 351us) client_negotiation.cc:240] Negotiation successful
0924 19:03:01.414217 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1071,"thread_start_us":781,"threads_started":1}
I20260924 19:03:01.416810 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:01.391606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47891 (local address 127.0.97.62:44897)
0924 19:03:01.394495 (+ 2889us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:01.394501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:01.394525 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:03:01.395547 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:01.395552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:01.395662 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:01.395786 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:01.397739 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:01.398949 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:01.401346 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:01.402069 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:01.402185 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:01.406723 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:01.406731 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:01.406736 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:01.406775 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:01.410503 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:01.411535 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:01.411540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:01.411544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:01.411698 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:01.412236 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:01.412241 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:01.412243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:01.412601 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:01.412765 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:01.416211 (+ 3446us) server_negotiation.cc:299] Negotiation successful
0924 19:03:01.416430 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2760,"thread_start_us":2645,"threads_started":1}
I20260924 19:03:01.418944 13610 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44897
I20260924 19:03:01.419335 13610 heartbeater.cc:464] Registering TS with master...
I20260924 19:03:01.420251 13610 heartbeater.cc:511] Master 127.0.97.62:44897 requested a full tablet report, sending...
I20260924 19:03:01.422293 13269 ts_manager.cc:194] Registered new tserver with Master: 12d14cd4fa0d4bbf9f1eee26a266fbd7 (127.0.97.2:42719)
I20260924 19:03:01.424166 13269 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47891
Sep 24 19:03:01 dist-test-slave-9fn1 krb5kdc[12258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 19:03:01.811255 13466 heartbeater.cc:464] Registering TS with master...
I20260924 19:03:01.811694 13466 heartbeater.cc:511] Master 127.0.97.62:44897 requested a full tablet report, sending...
I20260924 19:03:01.813028 13269 ts_manager.cc:194] Registered new tserver with Master: b52c4bfaf13b42d9b5feacb55cdc2780 (127.0.97.1:45819)
I20260924 19:03:01.814266 13269 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58289
W20260924 19:03:01.862468 13615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:03:01.862965 13615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:03:01.863170 13615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:03:01.874585 13615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:03:01.874779 13615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:03:01.874994 13615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:03:01.890066 13615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:33501/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:37785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=32859
--tserver_master_addrs=127.0.97.62:44897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:03:01.891953 13615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:03:01.893692 13615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:03:01.938093 13621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:01.942227 13620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:01.942710 13624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:03:01.943203 13615 server_base.cc:1034] running on GCE node
I20260924 19:03:01.944101 13615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:03:01.956149 13615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:03:01.987892 13615 hybrid_clock.cc:648] HybridClock initialized: now 1790276581985228 us; error 2637 us; skew 500 ppm
I20260924 19:03:01.988390 13615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:03:01 dist-test-slave-9fn1 krb5kdc[12258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:03:01.992657 13615 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 19:03:02.018076 13615 webserver.cc:460] Webserver started at http://127.0.97.3:32859/ using document root <none> and password file <none>
I20260924 19:03:02.019081 13615 fs_manager.cc:359] Metadata directory not provided
I20260924 19:03:02.019294 13615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 19:03:02 dist-test-slave-9fn1 krb5kdc[12258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 19:03:02.048971 13615 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 19:03:02.066032 13632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:02.068118 13615 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.002s sys 0.009s
I20260924 19:03:02.068346 13615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "b86ccf4c1df94b9fb5af66ccadcda0d0"
format_stamp: "Formatted at 2026-09-24 19:02:58 on dist-test-slave-9fn1"
server_key: "9508b06c89dca3303e6e775bdd1e049d"
server_key_iv: "ca8052dbffad51f1df4c2cedd353d600"
server_key_version: "kuduclusterkey@0"
I20260924 19:03:02.069778 13615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 24 19:03:02 dist-test-slave-9fn1 krb5kdc[12258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260924 19:03:02.167686 13615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:03:02.172650 13615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:03:02.172964 13615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:03:02.180778 13615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:03:02.188731 13615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:03:02.188848 13615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:02.188946 13615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:03:02.189008 13615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:02.428985 13610 heartbeater.cc:503] Master 127.0.97.62:44897 was elected leader, sending a full tablet report...
I20260924 19:03:02.450424 13615 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37785
I20260924 19:03:02.450474 13751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37785 every 8 connection(s)
I20260924 19:03:02.451982 13615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:03:02.461254 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 13615
Sep 24 19:03:02 dist-test-slave-9fn1 krb5kdc[12258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:03:02.481335 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:02.455047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46295 (local address 127.0.97.62:44897)
0924 19:03:02.458658 (+ 3611us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:02.458668 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:02.461234 (+ 2566us) server_negotiation.cc:407] Connection header received
0924 19:03:02.463586 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:02.463592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:02.463697 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:02.463793 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:02.465786 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.466926 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.469159 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.469800 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.469892 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:02.474239 (+ 4347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:02.474246 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:02.474251 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:02.474315 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:02.477823 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:02.478700 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:02.478705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:02.478709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:02.478863 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:02.479364 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:02.479368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:02.479372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:02.479704 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:02.479836 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:02.480721 (+ 885us) server_negotiation.cc:299] Negotiation successful
0924 19:03:02.480931 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3435,"thread_start_us":3269,"threads_started":1}
I20260924 19:03:02.483722 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:02.455949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44897 (local address 127.0.97.3:46295)
0924 19:03:02.460797 (+ 4848us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:02.460841 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:02.463176 (+ 2335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:02.464200 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:02.464211 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:02.464783 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:03:02.465587 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:02.465611 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.467112 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.467117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:02.469003 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:02.469017 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.470241 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.470248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:02.470527 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:02.471397 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:03:02.471425 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:03:02.473962 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:03:02.477980 (+ 4018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:02.477989 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:02.478006 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:02.478551 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:02.478998 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:02.479003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:02.479006 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:02.479236 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:02.479901 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:03:02.479909 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:03:02.480300 (+ 391us) client_negotiation.cc:769] Sending connection context
0924 19:03:02.481758 (+ 1458us) client_negotiation.cc:240] Negotiation successful
0924 19:03:02.482396 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4563,"thread_start_us":4311,"threads_started":1}
I20260924 19:03:02.485275 13752 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44897
I20260924 19:03:02.485599 13752 heartbeater.cc:464] Registering TS with master...
I20260924 19:03:02.486372 13752 heartbeater.cc:511] Master 127.0.97.62:44897 requested a full tablet report, sending...
I20260924 19:03:02.487913 13269 ts_manager.cc:194] Registered new tserver with Master: b86ccf4c1df94b9fb5af66ccadcda0d0 (127.0.97.3:37785)
I20260924 19:03:02.489068 13269 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:46295
I20260924 19:03:02.494069 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:02.462232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46062 (local address 127.0.97.62:44897)
0924 19:03:02.474326 (+ 12094us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:02.474332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:02.478002 (+ 3670us) server_negotiation.cc:407] Connection header received
0924 19:03:02.478444 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:02.478450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:02.478550 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:02.478624 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:02.480848 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.481999 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.483361 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.483824 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.483949 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:02.486861 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:02.486866 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:02.486869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:02.486896 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:02.489955 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:02.492056 (+ 2101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:02.492063 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:02.492067 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:02.492186 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:02.492559 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:02.492566 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:02.492569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:02.492887 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:02.493153 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:02.493450 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 19:03:02.493695 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11942,"thread_start_us":11790,"threads_started":1}
I20260924 19:03:02.496652 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:03:02.513396 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:02.499861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46064 (local address 127.0.97.62:44897)
0924 19:03:02.500055 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:02.500061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:02.500485 (+ 424us) server_negotiation.cc:407] Connection header received
0924 19:03:02.500613 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:02.500619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:02.500689 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:02.500786 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:02.502321 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.503373 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.504603 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.505191 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.505319 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:02.507706 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:02.507712 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:02.507717 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:02.507752 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:02.510874 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:02.511472 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:02.511478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:02.511482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:02.511623 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:02.511954 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:02.511959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:02.511963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:02.512273 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:02.512520 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:02.512845 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 19:03:02.513060 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 19:03:02.519104 13269 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:03:02.529752 13269 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:03:02.571288 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:02.544831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41026 (local address 127.0.97.3:37785)
0924 19:03:02.556580 (+ 11749us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:02.556594 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:02.561319 (+ 4725us) server_negotiation.cc:407] Connection header received
0924 19:03:02.562515 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:02.562524 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:02.562768 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:02.562922 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:03:02.563835 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.565334 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.568360 (+ 3026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.569814 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.569903 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:02.570130 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:02.570778 (+ 648us) server_negotiation.cc:299] Negotiation successful
0924 19:03:02.570943 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1127,"thread_start_us":943,"threads_started":1}
I20260924 19:03:02.572114 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:02.544495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37785 (local address 127.0.0.1:41026)
0924 19:03:02.561187 (+ 16692us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:02.561208 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:02.562313 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:02.563228 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:02.563233 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:02.563257 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:03:02.563639 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:02.563650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.565541 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.565546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:02.568183 (+ 2637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:02.568198 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.570326 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.570331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:02.570532 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:02.570577 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 19:03:02.571569 (+ 992us) client_negotiation.cc:240] Negotiation successful
0924 19:03:02.571757 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16554,"thread_start_us":514,"threads_started":1}
I20260924 19:03:02.572243 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:02.544199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42719 (local address 127.0.0.1:42476)
0924 19:03:02.545186 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:02.545222 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:02.545384 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:02.559686 (+ 14302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:02.559700 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:02.559751 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:03:02.560377 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:02.560396 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.566054 (+ 5658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.566063 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:02.568976 (+ 2913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:02.568988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.571390 (+ 2402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.571399 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:02.571567 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:02.571595 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 19:03:02.571659 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 19:03:02.571788 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":2,"thread_start_us":17162,"threads_started":1}
I20260924 19:03:02.574424 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:02.545337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42476 (local address 127.0.97.2:42719)
0924 19:03:02.557259 (+ 11922us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:02.557270 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:02.557312 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:03:02.557469 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:02.557482 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:02.558728 (+ 1246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:02.560739 (+ 2011us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:03:02.561039 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.563004 (+ 1965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.569617 (+ 6613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.571171 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.573628 (+ 2457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:02.573862 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:02.573929 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 19:03:02.574101 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11698,"thread_start_us":11493,"threads_started":1}
I20260924 19:03:02.576081 13682 tablet_service.cc:1467] Processing CreateTablet for tablet cfc9e96e5ea24d7f87979abaed38c4b5 (DEFAULT_TABLE table=test-table [id=f5b9683b31a64c77919686f823ac16c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:03:02.577555 13682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc9e96e5ea24d7f87979abaed38c4b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:03:02.578485 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:02.545796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45819 (local address 127.0.0.1:56154)
0924 19:03:02.569469 (+ 23673us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:02.569491 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:02.569669 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:02.570630 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:02.570635 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:02.570660 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:03:02.570990 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:02.571001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.573342 (+ 2341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.573347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:02.575405 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:02.575419 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.577708 (+ 2289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.577716 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:02.577887 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:02.577920 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 19:03:02.578009 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 19:03:02.578136 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23549,"thread_start_us":23506,"threads_started":1}
I20260924 19:03:02.578248 13545 tablet_service.cc:1467] Processing CreateTablet for tablet cfc9e96e5ea24d7f87979abaed38c4b5 (DEFAULT_TABLE table=test-table [id=f5b9683b31a64c77919686f823ac16c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:03:02.579756 13545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc9e96e5ea24d7f87979abaed38c4b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:03:02.587387 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:02.550514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56154 (local address 127.0.97.1:45819)
0924 19:03:02.565918 (+ 15404us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:02.565930 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:02.569793 (+ 3863us) server_negotiation.cc:407] Connection header received
0924 19:03:02.569896 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:02.569904 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:02.570155 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:02.570357 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:03:02.571200 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.573193 (+ 1993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.575612 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.577096 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.577219 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:02.577435 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:02.578906 (+ 1471us) server_negotiation.cc:299] Negotiation successful
0924 19:03:02.579077 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15199,"thread_start_us":14957,"threads_started":1}
I20260924 19:03:02.593387 13773 tablet_bootstrap.cc:492] T cfc9e96e5ea24d7f87979abaed38c4b5 P b86ccf4c1df94b9fb5af66ccadcda0d0: Bootstrap starting.
I20260924 19:03:02.594251 13401 tablet_service.cc:1467] Processing CreateTablet for tablet cfc9e96e5ea24d7f87979abaed38c4b5 (DEFAULT_TABLE table=test-table [id=f5b9683b31a64c77919686f823ac16c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:03:02.595705 13401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfc9e96e5ea24d7f87979abaed38c4b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:03:02.596594 13773 tablet_bootstrap.cc:654] T cfc9e96e5ea24d7f87979abaed38c4b5 P b86ccf4c1df94b9fb5af66ccadcda0d0: Neither blocks nor log segments found. Creating new log.
I20260924 19:03:02.597692 13773 log.cc:824] T cfc9e96e5ea24d7f87979abaed38c4b5 P b86ccf4c1df94b9fb5af66ccadcda0d0: Log is configured to *not* fsync() on all Append() calls
I20260924 19:03:02.599659 13775 tablet_bootstrap.cc:492] T cfc9e96e5ea24d7f87979abaed38c4b5 P 12d14cd4fa0d4bbf9f1eee26a266fbd7: Bootstrap starting.
I20260924 19:03:02.600507 13773 tablet_bootstrap.cc:492] T cfc9e96e5ea24d7f87979abaed38c4b5 P b86ccf4c1df94b9fb5af66ccadcda0d0: No bootstrap required, opened a new log
I20260924 19:03:02.600760 13773 ts_tablet_manager.cc:1397] T cfc9e96e5ea24d7f87979abaed38c4b5 P b86ccf4c1df94b9fb5af66ccadcda0d0: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 19:03:02.603106 13775 tablet_bootstrap.cc:654] T cfc9e96e5ea24d7f87979abaed38c4b5 P 12d14cd4fa0d4bbf9f1eee26a266fbd7: Neither blocks nor log segments found. Creating new log.
I20260924 19:03:02.604291 13775 log.cc:824] T cfc9e96e5ea24d7f87979abaed38c4b5 P 12d14cd4fa0d4bbf9f1eee26a266fbd7: Log is configured to *not* fsync() on all Append() calls
I20260924 19:03:02.607250 13775 tablet_bootstrap.cc:492] T cfc9e96e5ea24d7f87979abaed38c4b5 P 12d14cd4fa0d4bbf9f1eee26a266fbd7: No bootstrap required, opened a new log
I20260924 19:03:02.607503 13775 ts_tablet_manager.cc:1397] T cfc9e96e5ea24d7f87979abaed38c4b5 P 12d14cd4fa0d4bbf9f1eee26a266fbd7: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260924 19:03:02.608454 13773 raft_consensus.cc:348] T cfc9e96e5ea24d7f87979abaed38c4b5 P b86ccf4c1df94b9fb5af66ccadcda0d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d14cd4fa0d4bbf9f1eee26a266fbd7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42719 } } peers { permanent_uuid: "b86ccf4c1df94b9fb5af66ccadcda0d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37785 } } peers { permanent_uuid: "b52c4bfaf13b42d9b5feacb55cdc2780" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45819 } }
I20260924 19:03:02.608785 13773 raft_consensus.cc:374] T cfc9e96e5ea24d7f87979abaed38c4b5 P b86ccf4c1df94b9fb5af66ccadcda0d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:03:02.608917 13773 raft_consensus.cc:729] T cfc9e96e5ea24d7f87979abaed38c4b5 P b86ccf4c1df94b9fb5af66ccadcda0d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b86ccf4c1df94b9fb5af66ccadcda0d0, State: Initialized, Role: FOLLOWER
I20260924 19:03:02.609490 13773 consensus_queue.cc:260] T cfc9e96e5ea24d7f87979abaed38c4b5 P b86ccf4c1df94b9fb5af66ccadcda0d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d14cd4fa0d4bbf9f1eee26a266fbd7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42719 } } peers { permanent_uuid: "b86ccf4c1df94b9fb5af66ccadcda0d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37785 } } peers { permanent_uuid: "b52c4bfaf13b42d9b5feacb55cdc2780" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45819 } }
I20260924 19:03:02.610693 13752 heartbeater.cc:503] Master 127.0.97.62:44897 was elected leader, sending a full tablet report...
I20260924 19:03:02.611058 13773 ts_tablet_manager.cc:1428] T cfc9e96e5ea24d7f87979abaed38c4b5 P b86ccf4c1df94b9fb5af66ccadcda0d0: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260924 19:03:02.611943 13778 tablet_bootstrap.cc:492] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780: Bootstrap starting.
I20260924 19:03:02.615046 13778 tablet_bootstrap.cc:654] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780: Neither blocks nor log segments found. Creating new log.
I20260924 19:03:02.616254 13778 log.cc:824] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780: Log is configured to *not* fsync() on all Append() calls
I20260924 19:03:02.618736 13778 tablet_bootstrap.cc:492] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780: No bootstrap required, opened a new log
I20260924 19:03:02.618695 13775 raft_consensus.cc:348] T cfc9e96e5ea24d7f87979abaed38c4b5 P 12d14cd4fa0d4bbf9f1eee26a266fbd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d14cd4fa0d4bbf9f1eee26a266fbd7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42719 } } peers { permanent_uuid: "b86ccf4c1df94b9fb5af66ccadcda0d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37785 } } peers { permanent_uuid: "b52c4bfaf13b42d9b5feacb55cdc2780" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45819 } }
I20260924 19:03:02.618975 13778 ts_tablet_manager.cc:1397] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 19:03:02.619068 13775 raft_consensus.cc:374] T cfc9e96e5ea24d7f87979abaed38c4b5 P 12d14cd4fa0d4bbf9f1eee26a266fbd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:03:02.619164 13775 raft_consensus.cc:729] T cfc9e96e5ea24d7f87979abaed38c4b5 P 12d14cd4fa0d4bbf9f1eee26a266fbd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d14cd4fa0d4bbf9f1eee26a266fbd7, State: Initialized, Role: FOLLOWER
I20260924 19:03:02.619668 13775 consensus_queue.cc:260] T cfc9e96e5ea24d7f87979abaed38c4b5 P 12d14cd4fa0d4bbf9f1eee26a266fbd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d14cd4fa0d4bbf9f1eee26a266fbd7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42719 } } peers { permanent_uuid: "b86ccf4c1df94b9fb5af66ccadcda0d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37785 } } peers { permanent_uuid: "b52c4bfaf13b42d9b5feacb55cdc2780" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45819 } }
I20260924 19:03:02.621222 13775 ts_tablet_manager.cc:1428] T cfc9e96e5ea24d7f87979abaed38c4b5 P 12d14cd4fa0d4bbf9f1eee26a266fbd7: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260924 19:03:02.627876 13778 raft_consensus.cc:348] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d14cd4fa0d4bbf9f1eee26a266fbd7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42719 } } peers { permanent_uuid: "b86ccf4c1df94b9fb5af66ccadcda0d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37785 } } peers { permanent_uuid: "b52c4bfaf13b42d9b5feacb55cdc2780" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45819 } }
I20260924 19:03:02.628230 13778 raft_consensus.cc:374] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:03:02.628363 13778 raft_consensus.cc:729] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b52c4bfaf13b42d9b5feacb55cdc2780, State: Initialized, Role: FOLLOWER
I20260924 19:03:02.628861 13778 consensus_queue.cc:260] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d14cd4fa0d4bbf9f1eee26a266fbd7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42719 } } peers { permanent_uuid: "b86ccf4c1df94b9fb5af66ccadcda0d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37785 } } peers { permanent_uuid: "b52c4bfaf13b42d9b5feacb55cdc2780" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45819 } }
I20260924 19:03:02.629842 13466 heartbeater.cc:503] Master 127.0.97.62:44897 was elected leader, sending a full tablet report...
I20260924 19:03:02.630489 13778 ts_tablet_manager.cc:1428] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
W20260924 19:03:02.641475 13611 tablet.cc:2369] T cfc9e96e5ea24d7f87979abaed38c4b5 P 12d14cd4fa0d4bbf9f1eee26a266fbd7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:03:02.710240 13753 tablet.cc:2369] T cfc9e96e5ea24d7f87979abaed38c4b5 P b86ccf4c1df94b9fb5af66ccadcda0d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:03:02.770267 13467 tablet.cc:2369] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:03:02.814482 13781 raft_consensus.cc:482] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:03:02.814803 13781 raft_consensus.cc:504] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d14cd4fa0d4bbf9f1eee26a266fbd7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42719 } } peers { permanent_uuid: "b86ccf4c1df94b9fb5af66ccadcda0d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37785 } } peers { permanent_uuid: "b52c4bfaf13b42d9b5feacb55cdc2780" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45819 } }
I20260924 19:03:02.816510 13781 leader_election.cc:288] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12d14cd4fa0d4bbf9f1eee26a266fbd7 (127.0.97.2:42719), b86ccf4c1df94b9fb5af66ccadcda0d0 (127.0.97.3:37785)
I20260924 19:03:02.823065 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:02.816879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34905 (local address 127.0.97.2:42719)
0924 19:03:02.817153 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:02.817162 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:02.817711 (+ 549us) server_negotiation.cc:407] Connection header received
0924 19:03:02.817940 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:02.817944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:02.818044 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:02.818150 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:03:02.818695 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.819396 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.821278 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.822142 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.822307 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:02.822414 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:02.822700 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 19:03:02.822817 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 19:03:02.823148 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:02.816764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42719 (local address 127.0.97.1:34905)
0924 19:03:02.817599 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:02.817619 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:02.817792 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:02.818181 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:02.818185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:02.818211 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:03:02.818553 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:02.818567 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.819529 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.819535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:02.821109 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:02.821121 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.822349 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.822355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:02.822511 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:02.822537 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 19:03:02.822649 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 19:03:02.822775 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":2,"thread_start_us":893,"threads_started":1}
I20260924 19:03:02.823365 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:02.816920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57883 (local address 127.0.97.3:37785)
0924 19:03:02.817359 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:02.817368 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:02.818025 (+ 657us) server_negotiation.cc:407] Connection header received
0924 19:03:02.818213 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:02.818217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:02.818272 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:02.818368 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:03:02.818958 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.819634 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.821486 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.822353 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.822505 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:02.822610 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:02.822979 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 19:03:02.823088 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272}
I20260924 19:03:02.824059 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:02.816810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37785 (local address 127.0.97.1:57883)
0924 19:03:02.817925 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:02.817938 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:02.818044 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:02.818386 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:02.818390 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:02.818413 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:03:02.818810 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:02.818819 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.819763 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.819770 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:02.821345 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:02.821356 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.822585 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.822594 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:02.822786 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:02.822817 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 19:03:02.823481 (+ 664us) client_negotiation.cc:240] Negotiation successful
0924 19:03:02.823689 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1003,"mutex_wait_us":26,"thread_start_us":383,"threads_started":1}
I20260924 19:03:02.824524 13565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfc9e96e5ea24d7f87979abaed38c4b5" candidate_uuid: "b52c4bfaf13b42d9b5feacb55cdc2780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d14cd4fa0d4bbf9f1eee26a266fbd7" is_pre_election: true
I20260924 19:03:02.824998 13707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfc9e96e5ea24d7f87979abaed38c4b5" candidate_uuid: "b52c4bfaf13b42d9b5feacb55cdc2780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b86ccf4c1df94b9fb5af66ccadcda0d0" is_pre_election: true
I20260924 19:03:02.825117 13565 raft_consensus.cc:2454] T cfc9e96e5ea24d7f87979abaed38c4b5 P 12d14cd4fa0d4bbf9f1eee26a266fbd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b52c4bfaf13b42d9b5feacb55cdc2780 in term 0.
I20260924 19:03:02.825450 13707 raft_consensus.cc:2454] T cfc9e96e5ea24d7f87979abaed38c4b5 P b86ccf4c1df94b9fb5af66ccadcda0d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b52c4bfaf13b42d9b5feacb55cdc2780 in term 0.
I20260924 19:03:02.825795 13356 leader_election.cc:302] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12d14cd4fa0d4bbf9f1eee26a266fbd7, b52c4bfaf13b42d9b5feacb55cdc2780; no voters:
I20260924 19:03:02.826210 13781 raft_consensus.cc:2781] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:03:02.826355 13781 raft_consensus.cc:482] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:03:02.826476 13781 raft_consensus.cc:3037] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:03:02.828583 13781 raft_consensus.cc:504] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d14cd4fa0d4bbf9f1eee26a266fbd7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42719 } } peers { permanent_uuid: "b86ccf4c1df94b9fb5af66ccadcda0d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37785 } } peers { permanent_uuid: "b52c4bfaf13b42d9b5feacb55cdc2780" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45819 } }
I20260924 19:03:02.829166 13781 leader_election.cc:288] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780 [CANDIDATE]: Term 1 election: Requested vote from peers 12d14cd4fa0d4bbf9f1eee26a266fbd7 (127.0.97.2:42719), b86ccf4c1df94b9fb5af66ccadcda0d0 (127.0.97.3:37785)
I20260924 19:03:02.829613 13565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfc9e96e5ea24d7f87979abaed38c4b5" candidate_uuid: "b52c4bfaf13b42d9b5feacb55cdc2780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12d14cd4fa0d4bbf9f1eee26a266fbd7"
I20260924 19:03:02.829620 13707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfc9e96e5ea24d7f87979abaed38c4b5" candidate_uuid: "b52c4bfaf13b42d9b5feacb55cdc2780" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b86ccf4c1df94b9fb5af66ccadcda0d0"
I20260924 19:03:02.829824 13565 raft_consensus.cc:3037] T cfc9e96e5ea24d7f87979abaed38c4b5 P 12d14cd4fa0d4bbf9f1eee26a266fbd7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:03:02.829835 13707 raft_consensus.cc:3037] T cfc9e96e5ea24d7f87979abaed38c4b5 P b86ccf4c1df94b9fb5af66ccadcda0d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:03:02.832125 13707 raft_consensus.cc:2454] T cfc9e96e5ea24d7f87979abaed38c4b5 P b86ccf4c1df94b9fb5af66ccadcda0d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b52c4bfaf13b42d9b5feacb55cdc2780 in term 1.
I20260924 19:03:02.832296 13565 raft_consensus.cc:2454] T cfc9e96e5ea24d7f87979abaed38c4b5 P 12d14cd4fa0d4bbf9f1eee26a266fbd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b52c4bfaf13b42d9b5feacb55cdc2780 in term 1.
I20260924 19:03:02.832481 13354 leader_election.cc:302] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b52c4bfaf13b42d9b5feacb55cdc2780, b86ccf4c1df94b9fb5af66ccadcda0d0; no voters:
I20260924 19:03:02.832770 13781 raft_consensus.cc:2781] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:03:02.833236 13781 raft_consensus.cc:686] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780 [term 1 LEADER]: Becoming Leader. State: Replica: b52c4bfaf13b42d9b5feacb55cdc2780, State: Running, Role: LEADER
I20260924 19:03:02.833771 13781 consensus_queue.cc:237] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d14cd4fa0d4bbf9f1eee26a266fbd7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42719 } } peers { permanent_uuid: "b86ccf4c1df94b9fb5af66ccadcda0d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37785 } } peers { permanent_uuid: "b52c4bfaf13b42d9b5feacb55cdc2780" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45819 } }
I20260924 19:03:02.837999 13268 catalog_manager.cc:5495] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780 reported cstate change: term changed from 0 to 1, leader changed from <none> to b52c4bfaf13b42d9b5feacb55cdc2780 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "b52c4bfaf13b42d9b5feacb55cdc2780" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12d14cd4fa0d4bbf9f1eee26a266fbd7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b86ccf4c1df94b9fb5af66ccadcda0d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b52c4bfaf13b42d9b5feacb55cdc2780" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45819 } health_report { overall_health: HEALTHY } } }
I20260924 19:03:02.868554 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:02.863092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56156 (local address 127.0.97.1:45819)
0924 19:03:02.863291 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:02.863297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:02.863403 (+ 106us) server_negotiation.cc:407] Connection header received
0924 19:03:02.863673 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:02.863677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:02.863761 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:02.863915 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:03:02.864292 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.865001 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.866418 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.866829 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.866963 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:02.867453 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:03:02.867937 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:03:02.868098 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:02.868209 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 19:03:02.868282 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 19:03:02.876861 13565 raft_consensus.cc:1261] T cfc9e96e5ea24d7f87979abaed38c4b5 P 12d14cd4fa0d4bbf9f1eee26a266fbd7 [term 1 FOLLOWER]: Refusing update from remote peer b52c4bfaf13b42d9b5feacb55cdc2780: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:03:02.876861 13707 raft_consensus.cc:1261] T cfc9e96e5ea24d7f87979abaed38c4b5 P b86ccf4c1df94b9fb5af66ccadcda0d0 [term 1 FOLLOWER]: Refusing update from remote peer b52c4bfaf13b42d9b5feacb55cdc2780: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:03:02.877924 13784 consensus_queue.cc:1035] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b86ccf4c1df94b9fb5af66ccadcda0d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:03:02.878280 13781 consensus_queue.cc:1035] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12d14cd4fa0d4bbf9f1eee26a266fbd7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:03:02.888902 13787 mvcc.cc:205] Tried to move back new op lower bound from 7332972883452866560 to 7332972883290902528. Current Snapshot: MvccSnapshot[applied={T|T < 7332972883452866560}]
I20260924 19:03:02.889464 13788 mvcc.cc:205] Tried to move back new op lower bound from 7332972883452866560 to 7332972883290902528. Current Snapshot: MvccSnapshot[applied={T|T < 7332972883452866560}]
I20260924 19:03:02.903892 13269 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:03:02.904155 13269 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:03:02.907680 13269 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f7ea2d1a0e6429a87b1160d8f3d6746: Sending DeleteTablet for 3 replicas of tablet cfc9e96e5ea24d7f87979abaed38c4b5
I20260924 19:03:02.908850 13545 tablet_service.cc:1514] Processing DeleteTablet for tablet cfc9e96e5ea24d7f87979abaed38c4b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:03:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42476
I20260924 19:03:02.908875 13401 tablet_service.cc:1514] Processing DeleteTablet for tablet cfc9e96e5ea24d7f87979abaed38c4b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:03:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56154
I20260924 19:03:02.908939 13682 tablet_service.cc:1514] Processing DeleteTablet for tablet cfc9e96e5ea24d7f87979abaed38c4b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:03:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41026
I20260924 19:03:02.914548 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 13306
I20260924 19:03:02.930938 13797 tablet_replica.cc:330] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780: stopping tablet replica
I20260924 19:03:02.931524 13797 raft_consensus.cc:2229] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:03:02.932442 13797 raft_consensus.cc:2258] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:03:02.933269 13798 tablet_replica.cc:330] T cfc9e96e5ea24d7f87979abaed38c4b5 P 12d14cd4fa0d4bbf9f1eee26a266fbd7: stopping tablet replica
I20260924 19:03:02.933717 13798 raft_consensus.cc:2229] T cfc9e96e5ea24d7f87979abaed38c4b5 P 12d14cd4fa0d4bbf9f1eee26a266fbd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:03:02.934227 13798 raft_consensus.cc:2258] T cfc9e96e5ea24d7f87979abaed38c4b5 P 12d14cd4fa0d4bbf9f1eee26a266fbd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:03:02.936429 13798 ts_tablet_manager.cc:1905] T cfc9e96e5ea24d7f87979abaed38c4b5 P 12d14cd4fa0d4bbf9f1eee26a266fbd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:03:02.938982 13797 ts_tablet_manager.cc:1905] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:03:02.942463 13799 tablet_replica.cc:330] T cfc9e96e5ea24d7f87979abaed38c4b5 P b86ccf4c1df94b9fb5af66ccadcda0d0: stopping tablet replica
I20260924 19:03:02.942901 13799 raft_consensus.cc:2229] T cfc9e96e5ea24d7f87979abaed38c4b5 P b86ccf4c1df94b9fb5af66ccadcda0d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:03:02.943363 13799 raft_consensus.cc:2258] T cfc9e96e5ea24d7f87979abaed38c4b5 P b86ccf4c1df94b9fb5af66ccadcda0d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:03:02.945428 13799 ts_tablet_manager.cc:1905] T cfc9e96e5ea24d7f87979abaed38c4b5 P b86ccf4c1df94b9fb5af66ccadcda0d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:03:02.947613 13798 ts_tablet_manager.cc:1918] T cfc9e96e5ea24d7f87979abaed38c4b5 P 12d14cd4fa0d4bbf9f1eee26a266fbd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:03:02.947779 13798 log.cc:1196] T cfc9e96e5ea24d7f87979abaed38c4b5 P 12d14cd4fa0d4bbf9f1eee26a266fbd7: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-1/wal/wals/cfc9e96e5ea24d7f87979abaed38c4b5
I20260924 19:03:02.948336 13798 ts_tablet_manager.cc:1939] T cfc9e96e5ea24d7f87979abaed38c4b5 P 12d14cd4fa0d4bbf9f1eee26a266fbd7: Deleting consensus metadata
I20260924 19:03:02.949064 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:02.916428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56166 (local address 127.0.97.1:45819)
0924 19:03:02.916652 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:02.916660 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:02.916683 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:03:02.916754 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:02.916762 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:02.916857 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:02.916986 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:02.926230 (+ 9244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.927840 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.930503 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:02.931427 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:02.931607 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:02.936416 (+ 4809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:02.936436 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:02.936459 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:02.936541 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:02.942471 (+ 5930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:02.943868 (+ 1397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:02.943876 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:02.943879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:02.944080 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:02.945954 (+ 1874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:02.945962 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:02.945968 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:02.946424 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:02.946739 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:02.948429 (+ 1690us) server_negotiation.cc:299] Negotiation successful
0924 19:03:02.948661 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 19:03:02.949870 13461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:03:02.949856 13797 ts_tablet_manager.cc:1918] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:03:02.950069 13797 log.cc:1196] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-0/wal/wals/cfc9e96e5ea24d7f87979abaed38c4b5
I20260924 19:03:02.950213 13255 catalog_manager.cc:4841] TS 12d14cd4fa0d4bbf9f1eee26a266fbd7 (127.0.97.2:42719): tablet cfc9e96e5ea24d7f87979abaed38c4b5 (table test-table [id=f5b9683b31a64c77919686f823ac16c5]) successfully deleted
I20260924 19:03:02.953893 13799 ts_tablet_manager.cc:1918] T cfc9e96e5ea24d7f87979abaed38c4b5 P b86ccf4c1df94b9fb5af66ccadcda0d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:03:02.954071 13799 log.cc:1196] T cfc9e96e5ea24d7f87979abaed38c4b5 P b86ccf4c1df94b9fb5af66ccadcda0d0: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276386554690-388-0/minicluster-data/ts-2/wal/wals/cfc9e96e5ea24d7f87979abaed38c4b5
I20260924 19:03:02.954604 13799 ts_tablet_manager.cc:1939] T cfc9e96e5ea24d7f87979abaed38c4b5 P b86ccf4c1df94b9fb5af66ccadcda0d0: Deleting consensus metadata
I20260924 19:03:02.956360 13253 catalog_manager.cc:4841] TS b86ccf4c1df94b9fb5af66ccadcda0d0 (127.0.97.3:37785): tablet cfc9e96e5ea24d7f87979abaed38c4b5 (table test-table [id=f5b9683b31a64c77919686f823ac16c5]) successfully deleted
I20260924 19:03:03.046701 13797 ts_tablet_manager.cc:1939] T cfc9e96e5ea24d7f87979abaed38c4b5 P b52c4bfaf13b42d9b5feacb55cdc2780: Deleting consensus metadata
I20260924 19:03:03.046718 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 13306
W20260924 19:03:03.064322 13253 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:45819 (error 108)
I20260924 19:03:03.064561 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 13470
I20260924 19:03:03.066192 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:03.065702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45819 (local address 127.0.0.1:56172)
0924 19:03:03.065888 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:03.066022 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 19:03:03.066411 13253 catalog_manager.cc:4584] TS b52c4bfaf13b42d9b5feacb55cdc2780 (127.0.97.1:45819): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cfc9e96e5ea24d7f87979abaed38c4b5: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45819: connect: Connection refused (error 111)
I20260924 19:03:03.082327 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:03.065341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42484 (local address 127.0.97.2:42719)
0924 19:03:03.065544 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:03.065552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:03.065573 (+ 21us) server_negotiation.cc:407] Connection header received
0924 19:03:03.065743 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:03.065748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:03.065840 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:03.065932 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:03.067361 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:03.068085 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:03.069285 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:03.069744 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:03.069836 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:03.076609 (+ 6773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:03.076622 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:03.076636 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:03.076667 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:03.079608 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:03.080273 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:03.080281 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:03.080283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:03.080412 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:03.080781 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:03.080787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:03.080791 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:03.081138 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:03.081467 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:03.081769 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 19:03:03.081966 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 19:03:03.082998 13605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:03:03.119884 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:03.119346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45819 (local address 127.0.0.1:56186)
0924 19:03:03.119595 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:03.119715 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260924 19:03:03.120990 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:03.120563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45819 (local address 127.0.0.1:56188)
0924 19:03:03.120711 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:03.120838 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 19:03:03.171222 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 13470
I20260924 19:03:03.181404 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:03.180917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45819 (local address 127.0.0.1:56198)
0924 19:03:03.181124 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:03.181203 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 19:03:03.182592 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:03.182257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45819 (local address 127.0.0.1:56206)
0924 19:03:03.182406 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:03.182489 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 19:03:03.188053 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 13615
I20260924 19:03:03.200832 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:03.188799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41028 (local address 127.0.97.3:37785)
0924 19:03:03.188960 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:03.188968 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:03.189004 (+ 36us) server_negotiation.cc:407] Connection header received
0924 19:03:03.189225 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:03.189229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:03.189305 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:03.189411 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:03.190629 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:03.191327 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:03.192534 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:03.192930 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:03.193059 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:03.195535 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:03.195565 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:03.195582 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:03.195612 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:03.198171 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:03.198843 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:03.198859 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:03.198861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:03.198981 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:03.199367 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:03.199373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:03.199381 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:03.199706 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:03.200016 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:03.200284 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 19:03:03.200483 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 19:03:03.201550 13747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:03:03.263360 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:03.262847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45819 (local address 127.0.0.1:56208)
0924 19:03:03.263072 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:03.263175 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 19:03:03.264386 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:03.264019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45819 (local address 127.0.0.1:56212)
0924 19:03:03.264207 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:03.264269 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:45819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260924 19:03:03.289265 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 13615
I20260924 19:03:03.305938 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 13236
I20260924 19:03:03.307125 13299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:03:03.416468 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 13236
I20260924 19:03:03.440783 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 19:03:03.775987 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 19:03:03.776214 388 mini_ranger.cc:66] Stopping Ranger...
I20260924 19:03:04.120613 388 mini_ranger.cc:68] Stopped Ranger
2026-09-24 19:03:04.120 UTC [12288] LOG: received smart shutdown request
2026-09-24 19:03:04.123 UTC [12288] LOG: background worker "logical replication launcher" (PID 12297) exited with exit code 1
2026-09-24 19:03:04.124 UTC [12292] LOG: shutting down
2026-09-24 19:03:04.295 UTC [12288] LOG: database system is shut down
2026-09-24T19:03:04Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52960 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:03:04 dist-test-slave-9fn1 krb5kdc[13813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:03:04 dist-test-slave-9fn1 krb5kdc[13813](info): set up 2 sockets
Sep 24 19:03:04 dist-test-slave-9fn1 krb5kdc[13813](info): commencing operation
krb5kdc: starting...
W20260924 19:03:05.521829 388 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:03:05 dist-test-slave-9fn1 krb5kdc[13813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:03:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:03:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 19:03:05.747392 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46223
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46223,127.0.97.61:40795,127.0.97.60:44917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:03:06.070715 13826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:03:06.071017 13826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:03:06.071108 13826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:03:06.078646 13826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:03:06.078765 13826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:03:06.078821 13826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:03:06.078884 13826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:03:06.078928 13826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:03:06.088737 13826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46223,127.0.97.61:40795,127.0.97.60:44917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:03:06.090004 13826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:03:06.091241 13826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:03:06.102840 13831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:06.102877 13834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:06.102957 13832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:03:06.104123 13826 server_base.cc:1034] running on GCE node
I20260924 19:03:06.104950 13826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:03:06.106412 13826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:03:06.107641 13826 hybrid_clock.cc:648] HybridClock initialized: now 1790276586107606 us; error 56 us; skew 500 ppm
I20260924 19:03:06.107992 13826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:03:06 dist-test-slave-9fn1 krb5kdc[13813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:03:06.111219 13826 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:03:06.112769 13826 webserver.cc:460] Webserver started at http://127.0.97.62:40521/ using document root <none> and password file <none>
I20260924 19:03:06.113468 13826 fs_manager.cc:359] Metadata directory not provided
I20260924 19:03:06.113683 13826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:03:06.113987 13826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:03:06.116897 13826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "2537c653c6a848abafdeaa5db27e6b5e"
format_stamp: "Formatted at 2026-09-24 19:03:06 on dist-test-slave-9fn1"
server_key: "9dfadeab9df9827502bb0f8f7e8f550f"
server_key_iv: "b70d4aa007704ef4d28535f045328ce6"
server_key_version: "encryptionkey@0"
I20260924 19:03:06.117786 13826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "2537c653c6a848abafdeaa5db27e6b5e"
format_stamp: "Formatted at 2026-09-24 19:03:06 on dist-test-slave-9fn1"
server_key: "9dfadeab9df9827502bb0f8f7e8f550f"
server_key_iv: "b70d4aa007704ef4d28535f045328ce6"
server_key_version: "encryptionkey@0"
I20260924 19:03:06.122874 13826 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 19:03:06.126235 13841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:06.127553 13826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:03:06.127743 13826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "2537c653c6a848abafdeaa5db27e6b5e"
format_stamp: "Formatted at 2026-09-24 19:03:06 on dist-test-slave-9fn1"
server_key: "9dfadeab9df9827502bb0f8f7e8f550f"
server_key_iv: "b70d4aa007704ef4d28535f045328ce6"
server_key_version: "encryptionkey@0"
I20260924 19:03:06.127952 13826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:03:06.142047 13826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:03:06.146360 13826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:03:06.146646 13826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:03:06.165442 13826 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46223
I20260924 19:03:06.165477 13892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46223 every 8 connection(s)
I20260924 19:03:06.166787 13826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:03:06.168534 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 13826
I20260924 19:03:06.168761 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:03:06.169605 388 external_mini_cluster.cc:1428] Setting key b7d0f481b7d3a85f289125a554a57f25
I20260924 19:03:06.171317 13893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:03:06.180208 13893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46223 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 24 19:03:06 dist-test-slave-9fn1 krb5kdc[13813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:03:06.202502 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.184831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58736 (local address 127.0.97.62:46223)
0924 19:03:06.185636 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:06.185644 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:06.185710 (+ 66us) server_negotiation.cc:407] Connection header received
0924 19:03:06.186819 (+ 1109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:06.186825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:06.187047 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:06.187259 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:06.188786 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:06.190056 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:06.191411 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:06.191972 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:06.192172 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:06.195406 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:06.195414 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:06.195424 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:06.195451 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:06.198184 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:06.198863 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:06.198872 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:06.198875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:06.199017 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:06.199414 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:06.199419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:06.199424 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:06.199907 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:06.200163 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:06.200677 (+ 514us) server_negotiation.cc:299] Negotiation successful
0924 19:03:06.201040 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":586,"thread_start_us":401,"threads_started":1}
I20260924 19:03:06.202525 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.183544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46223 (local address 127.0.0.1:58736)
0924 19:03:06.184335 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:06.184374 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:06.186441 (+ 2067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:06.187280 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:06.187287 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:06.188043 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:03:06.188597 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:06.188615 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:06.190210 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:06.190220 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:06.191276 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:06.191286 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:06.192204 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:06.192210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:06.192399 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:06.193018 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:03:06.193044 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:03:06.195153 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:03:06.198330 (+ 3177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:06.198342 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:06.198353 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:06.198737 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:06.199136 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:06.199141 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:06.199144 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:06.199298 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:06.200101 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:03:06.200109 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:03:06.200325 (+ 216us) client_negotiation.cc:769] Sending connection context
0924 19:03:06.200566 (+ 241us) client_negotiation.cc:240] Negotiation successful
0924 19:03:06.201147 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":228,"thread_start_us":340,"threads_started":1}
I20260924 19:03:06.205718 13893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40795 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:03:06.206815 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.206450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40795 (local address 127.0.0.1:53694)
0924 19:03:06.206603 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:06.206705 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 19:03:06.207106 13842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40795: connect: Connection refused (error 111)
I20260924 19:03:06.208657 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.208312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40795 (local address 127.0.0.1:53698)
0924 19:03:06.208464 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:06.208539 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 19:03:06.208876 13893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40795: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40795: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260924 19:03:06.221639 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40795
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46223,127.0.97.61:40795,127.0.97.60:44917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:03:06.258172 13893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40795 } attempt: 1
I20260924 19:03:06.259356 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.258913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40795 (local address 127.0.0.1:53706)
0924 19:03:06.259112 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:06.259226 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 19:03:06.260468 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.260106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40795 (local address 127.0.0.1:53712)
0924 19:03:06.260277 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:06.260348 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260924 19:03:06.260761 13893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40795: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40795: connect: Connection refused (error 111)
I20260924 19:03:06.329016 13893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40795 } attempt: 2
I20260924 19:03:06.330294 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.329798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40795 (local address 127.0.0.1:53718)
0924 19:03:06.330041 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:06.330138 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 19:03:06.331285 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.330927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40795 (local address 127.0.0.1:53722)
0924 19:03:06.331072 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:06.331161 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 19:03:06.331477 13893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40795: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40795: connect: Connection refused (error 111)
I20260924 19:03:06.422758 13893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40795 } attempt: 3
I20260924 19:03:06.423882 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.423441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40795 (local address 127.0.0.1:53732)
0924 19:03:06.423652 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:06.423752 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 19:03:06.424878 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.424536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40795 (local address 127.0.0.1:53744)
0924 19:03:06.424676 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:06.424760 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 19:03:06.425120 13893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40795: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40795: connect: Connection refused (error 111)
W20260924 19:03:06.540669 13900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:03:06.541029 13900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:03:06.541196 13900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:03:06.548205 13900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:03:06.548309 13900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:03:06.548422 13900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:03:06.548518 13900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:03:06.548604 13900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:03:06.558358 13900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46223,127.0.97.61:40795,127.0.97.60:44917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:40795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:03:06.559650 13900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:03:06.560946 13900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:03:06.568352 13893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40795 } attempt: 4
I20260924 19:03:06.569465 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.568987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40795 (local address 127.0.0.1:53754)
0924 19:03:06.569261 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:06.569342 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260924 19:03:06.570487 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.570172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40795 (local address 127.0.0.1:53764)
0924 19:03:06.570295 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:06.570377 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 19:03:06.570730 13893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:40795: Network error: Client connection negotiation failed: client connection to 127.0.97.61:40795: connect: Connection refused (error 111)
W20260924 19:03:06.574163 13908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:06.574282 13905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:06.574321 13906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:03:06.575234 13900 server_base.cc:1034] running on GCE node
I20260924 19:03:06.575908 13900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:03:06.577150 13900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:03:06.578375 13900 hybrid_clock.cc:648] HybridClock initialized: now 1790276586578341 us; error 45 us; skew 500 ppm
I20260924 19:03:06.578727 13900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:03:06 dist-test-slave-9fn1 krb5kdc[13813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:03:06.587961 13900 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260924 19:03:06.589572 13900 webserver.cc:460] Webserver started at http://127.0.97.61:41223/ using document root <none> and password file <none>
I20260924 19:03:06.590263 13900 fs_manager.cc:359] Metadata directory not provided
I20260924 19:03:06.590410 13900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:03:06.590751 13900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:03:06.593747 13900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-1/data/instance:
uuid: "a6a4ebe2dae640e7bea7465915f5012c"
format_stamp: "Formatted at 2026-09-24 19:03:06 on dist-test-slave-9fn1"
server_key: "58d890ae5c32d078aa81675a9272e98a"
server_key_iv: "2c494ffd44891fd6b6423e40b84a4c64"
server_key_version: "encryptionkey@0"
I20260924 19:03:06.594542 13900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-1/wal/instance:
uuid: "a6a4ebe2dae640e7bea7465915f5012c"
format_stamp: "Formatted at 2026-09-24 19:03:06 on dist-test-slave-9fn1"
server_key: "58d890ae5c32d078aa81675a9272e98a"
server_key_iv: "2c494ffd44891fd6b6423e40b84a4c64"
server_key_version: "encryptionkey@0"
I20260924 19:03:06.599551 13900 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:03:06.602869 13915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:06.604158 13900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:03:06.604331 13900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-1/wal
uuid: "a6a4ebe2dae640e7bea7465915f5012c"
format_stamp: "Formatted at 2026-09-24 19:03:06 on dist-test-slave-9fn1"
server_key: "58d890ae5c32d078aa81675a9272e98a"
server_key_iv: "2c494ffd44891fd6b6423e40b84a4c64"
server_key_version: "encryptionkey@0"
I20260924 19:03:06.604538 13900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:03:06.630142 13900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:03:06.634334 13900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:03:06.634620 13900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:03:06.652609 13966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:40795 every 8 connection(s)
I20260924 19:03:06.652616 13900 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:40795
I20260924 19:03:06.654014 13900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-1/data/info.pb
I20260924 19:03:06.657763 13967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:03:06.663000 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 13900
I20260924 19:03:06.663190 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-1/wal/instance
I20260924 19:03:06.663631 388 external_mini_cluster.cc:1428] Setting key 72f2ba847618fa5280ab4d70b858c3a0
I20260924 19:03:06.664403 13967 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46223 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:03:06 dist-test-slave-9fn1 krb5kdc[13813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260924 19:03:06.684070 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.667069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58748 (local address 127.0.97.62:46223)
0924 19:03:06.667285 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:06.667292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:06.668579 (+ 1287us) server_negotiation.cc:407] Connection header received
0924 19:03:06.670490 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:06.670495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:06.670585 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:06.670703 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:06.672172 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:06.672834 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:06.674420 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:06.674839 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:06.674976 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:06.678385 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:06.678389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:06.678393 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:06.678424 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:06.681018 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:06.681838 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:06.681842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:06.681844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:06.681960 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:06.682368 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:06.682373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:06.682378 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:06.682661 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:06.682825 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:06.683515 (+ 690us) server_negotiation.cc:299] Negotiation successful
0924 19:03:06.683731 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 19:03:06.685024 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.667589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46223 (local address 127.0.0.1:58748)
0924 19:03:06.668400 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:06.668450 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:06.670222 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:06.670931 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:06.670939 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:06.671430 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:03:06.671970 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:06.671986 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:06.673012 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:06.673016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:06.674277 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:06.674288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:06.675064 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:06.675071 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:06.675300 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:06.675991 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:03:06.676012 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:03:06.678166 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:03:06.681164 (+ 2998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:06.681175 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:06.681190 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:06.681705 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:06.682071 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:06.682076 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:06.682080 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:06.682247 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:06.682799 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:03:06.682807 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:03:06.683158 (+ 351us) client_negotiation.cc:769] Sending connection context
0924 19:03:06.683413 (+ 255us) client_negotiation.cc:240] Negotiation successful
0924 19:03:06.683907 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":367,"threads_started":1}
I20260924 19:03:06.686820 13967 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40795 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:03:06 dist-test-slave-9fn1 krb5kdc[13813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 19:03:06.700819 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.687603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40795 (local address 127.0.0.1:53780)
0924 19:03:06.687790 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:06.687809 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:06.687908 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:06.688866 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:06.688871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:06.689277 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:03:06.689402 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:06.689409 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:06.690734 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:06.690742 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:06.691582 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:06.691593 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:06.692411 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:06.692417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:06.692598 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:06.693160 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:03:06.693176 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:03:06.695135 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:03:06.698361 (+ 3226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:06.698368 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:06.698374 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:06.698727 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:06.699086 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:06.699091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:06.699094 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:06.699254 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:06.699840 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:03:06.699847 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:03:06.700002 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 19:03:06.700137 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 19:03:06.700417 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 19:03:06.700920 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.687877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53780 (local address 127.0.97.61:40795)
0924 19:03:06.688488 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:06.688494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:06.688517 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:03:06.688555 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:06.688561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:06.688727 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:06.688884 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:06.689571 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:06.690580 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:06.691723 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:06.692252 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:06.692395 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:06.695356 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:06.695367 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:06.695379 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:06.695405 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:06.698238 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:06.698857 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:06.698870 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:06.698872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:06.698975 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:06.699354 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:06.699361 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:06.699364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:06.699709 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:06.699927 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:06.700283 (+ 356us) server_negotiation.cc:299] Negotiation successful
0924 19:03:06.700531 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":474,"thread_start_us":341,"threads_started":1}
I20260924 19:03:06.703332 13967 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:03:06.704358 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.703918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44917 (local address 127.0.0.1:35040)
0924 19:03:06.704096 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:06.704223 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 19:03:06.704627 13917 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44917: connect: Connection refused (error 111)
I20260924 19:03:06.706137 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.705820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44917 (local address 127.0.0.1:35044)
0924 19:03:06.705956 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:06.706038 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 19:03:06.706400 13967 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44917: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44917: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260924 19:03:06.710578 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44917
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46223,127.0.97.61:40795,127.0.97.60:44917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:03:06.755733 13967 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } attempt: 1
I20260924 19:03:06.757162 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.756501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44917 (local address 127.0.0.1:35052)
0924 19:03:06.756766 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:06.756919 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 19:03:06.758308 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.757845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44917 (local address 127.0.0.1:35068)
0924 19:03:06.758049 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:06.758152 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 19:03:06.758607 13967 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44917: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44917: connect: Connection refused (error 111)
I20260924 19:03:06.826887 13967 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } attempt: 2
I20260924 19:03:06.828117 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.827592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44917 (local address 127.0.0.1:35078)
0924 19:03:06.827816 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:06.827929 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 19:03:06.829141 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.828771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44917 (local address 127.0.0.1:35090)
0924 19:03:06.828917 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:06.829001 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 19:03:06.829355 13967 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44917: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44917: connect: Connection refused (error 111)
I20260924 19:03:06.870016 13893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40795 } attempt: 5
Sep 24 19:03:06 dist-test-slave-9fn1 krb5kdc[13813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 19:03:06.885179 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.870894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53782 (local address 127.0.97.61:40795)
0924 19:03:06.871055 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:06.871059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:06.871079 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:03:06.871282 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:06.871286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:06.871346 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:06.871440 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:06.872455 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:06.873143 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:06.875124 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:06.875601 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:06.875751 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:06.879903 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:06.879908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:06.879912 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:06.879947 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:06.882576 (+ 2629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:06.883353 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:06.883357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:06.883359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:06.883481 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:06.883875 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:06.883878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:06.883880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:06.884146 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:06.884253 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:06.884790 (+ 537us) server_negotiation.cc:299] Negotiation successful
0924 19:03:06.884954 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 19:03:06.885373 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.870755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40795 (local address 127.0.0.1:53782)
0924 19:03:06.870978 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:06.871002 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:06.871134 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:06.871499 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:06.871504 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:06.872024 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:03:06.872266 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:06.872283 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:06.873300 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:06.873308 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:06.874923 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:06.874953 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:06.875780 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:06.875788 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:06.876038 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:06.876758 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:03:06.876789 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:03:06.879697 (+ 2908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:03:06.882720 (+ 3023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:06.882726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:06.882730 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:06.883245 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:06.883592 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:06.883597 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:06.883600 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:06.883772 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:06.884274 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:03:06.884282 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:03:06.884485 (+ 203us) client_negotiation.cc:769] Sending connection context
0924 19:03:06.884697 (+ 212us) client_negotiation.cc:240] Negotiation successful
0924 19:03:06.885036 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 19:03:06.886677 13893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:03:06.888032 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.887478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44917 (local address 127.0.0.1:35092)
0924 19:03:06.887712 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:06.887828 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 19:03:06.889145 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.888699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44917 (local address 127.0.0.1:35102)
0924 19:03:06.888875 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:06.888999 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 19:03:06.889398 13893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44917: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44917: connect: Connection refused (error 111)
I20260924 19:03:06.920610 13967 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } attempt: 3
I20260924 19:03:06.921877 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.921305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44917 (local address 127.0.0.1:35116)
0924 19:03:06.921580 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:06.921688 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 19:03:06.923002 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.922505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44917 (local address 127.0.0.1:35118)
0924 19:03:06.922729 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:06.922833 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260924 19:03:06.923237 13967 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44917: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44917: connect: Connection refused (error 111)
I20260924 19:03:06.940610 13893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } attempt: 1
I20260924 19:03:06.941843 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.941225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44917 (local address 127.0.0.1:35120)
0924 19:03:06.941466 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:06.941625 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 19:03:06.942919 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:06.942537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44917 (local address 127.0.0.1:35130)
0924 19:03:06.942707 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:06.942791 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 19:03:06.943138 13893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44917: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44917: connect: Connection refused (error 111)
I20260924 19:03:07.011386 13893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } attempt: 2
I20260924 19:03:07.012701 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:07.012080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44917 (local address 127.0.0.1:35134)
0924 19:03:07.012349 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:07.012470 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 19:03:07.013780 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:07.013362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44917 (local address 127.0.0.1:35142)
0924 19:03:07.013537 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:07.013634 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 19:03:07.014046 13893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44917: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44917: connect: Connection refused (error 111)
W20260924 19:03:07.036731 13974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:03:07.037014 13974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:03:07.037106 13974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:03:07.044162 13974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:03:07.044268 13974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:03:07.044320 13974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:03:07.044374 13974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:03:07.044409 13974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:03:07.053911 13974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46223,127.0.97.61:40795,127.0.97.60:44917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:44917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:03:07.055085 13974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:03:07.056280 13974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:03:07.066445 13967 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } attempt: 4
W20260924 19:03:07.067380 13979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:03:07.067736 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:07.067199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44917 (local address 127.0.0.1:35150)
0924 19:03:07.067435 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:07.067551 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260924 19:03:07.068039 13982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:07.068344 13980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:03:07.068611 13974 server_base.cc:1034] running on GCE node
I20260924 19:03:07.069090 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:07.068559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44917 (local address 127.0.0.1:35158)
0924 19:03:07.068865 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:07.068950 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
W20260924 19:03:07.069360 13967 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44917: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44917: connect: Connection refused (error 111)
I20260924 19:03:07.069598 13974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:03:07.070851 13974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:03:07.072053 13974 hybrid_clock.cc:648] HybridClock initialized: now 1790276587072008 us; error 23 us; skew 500 ppm
I20260924 19:03:07.072336 13974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:03:07 dist-test-slave-9fn1 krb5kdc[13813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:03:07.075264 13974 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260924 19:03:07.076565 13974 webserver.cc:460] Webserver started at http://127.0.97.60:40353/ using document root <none> and password file <none>
I20260924 19:03:07.077193 13974 fs_manager.cc:359] Metadata directory not provided
I20260924 19:03:07.077287 13974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:03:07.077597 13974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:03:07.080315 13974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-2/data/instance:
uuid: "6b8d5056c9a944d88bdacfb97edbb0b3"
format_stamp: "Formatted at 2026-09-24 19:03:07 on dist-test-slave-9fn1"
server_key: "ac92c6d577cb7970a5b372038d4d2492"
server_key_iv: "c27381db11c0e7f9d7e8ab3f9831c570"
server_key_version: "encryptionkey@0"
I20260924 19:03:07.081039 13974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-2/wal/instance:
uuid: "6b8d5056c9a944d88bdacfb97edbb0b3"
format_stamp: "Formatted at 2026-09-24 19:03:07 on dist-test-slave-9fn1"
server_key: "ac92c6d577cb7970a5b372038d4d2492"
server_key_iv: "c27381db11c0e7f9d7e8ab3f9831c570"
server_key_version: "encryptionkey@0"
I20260924 19:03:07.085857 13974 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260924 19:03:07.088937 13989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:07.090386 13974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:03:07.090518 13974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-2/wal
uuid: "6b8d5056c9a944d88bdacfb97edbb0b3"
format_stamp: "Formatted at 2026-09-24 19:03:07 on dist-test-slave-9fn1"
server_key: "ac92c6d577cb7970a5b372038d4d2492"
server_key_iv: "c27381db11c0e7f9d7e8ab3f9831c570"
server_key_version: "encryptionkey@0"
I20260924 19:03:07.090691 13974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:03:07.110837 13974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:03:07.114959 13974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:03:07.115227 13974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:03:07.120301 13893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } attempt: 3
I20260924 19:03:07.121634 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:07.120993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44917 (local address 127.0.0.1:35160)
0924 19:03:07.121245 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:07.121363 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 19:03:07.122820 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:07.122343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44917 (local address 127.0.0.1:35164)
0924 19:03:07.122595 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:07.122692 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260924 19:03:07.123065 13893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:44917: Network error: Client connection negotiation failed: client connection to 127.0.97.60:44917: connect: Connection refused (error 111)
I20260924 19:03:07.133388 13974 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:44917
I20260924 19:03:07.133395 14040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:44917 every 8 connection(s)
I20260924 19:03:07.134744 13974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-2/data/info.pb
I20260924 19:03:07.138559 14041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:03:07.140012 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 13974
I20260924 19:03:07.140185 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276386554690-388-0/minicluster-data/master-2/wal/instance
I20260924 19:03:07.140655 388 external_mini_cluster.cc:1428] Setting key 86b8ecff5de1535a8f995829a7670eb8
I20260924 19:03:07.148304 14041 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46223 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:03:07 dist-test-slave-9fn1 krb5kdc[13813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 24 19:03:07 dist-test-slave-9fn1 krb5kdc[13813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:03:07.165650 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:07.142639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35166 (local address 127.0.97.60:44917)
0924 19:03:07.143452 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:07.143467 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:07.143519 (+ 52us) server_negotiation.cc:407] Connection header received
0924 19:03:07.145217 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:07.145251 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:07.145815 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:07.146277 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:07.147501 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.149559 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.151166 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.152326 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.152505 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:07.157339 (+ 4834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:07.157349 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:07.157365 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:07.157414 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:07.160971 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:07.161689 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:07.161698 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:07.161708 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:07.161904 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:07.162357 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:07.162364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:07.162368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:07.163157 (+ 789us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:07.163446 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:07.163882 (+ 436us) server_negotiation.cc:299] Negotiation successful
0924 19:03:07.164337 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":352,"threads_started":1}
I20260924 19:03:07.167695 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:07.151203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58750 (local address 127.0.97.62:46223)
0924 19:03:07.151456 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:07.151464 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:07.152079 (+ 615us) server_negotiation.cc:407] Connection header received
0924 19:03:07.152257 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:07.152261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:07.152322 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:07.152402 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:07.154084 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.155211 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.156568 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.157247 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.157340 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:07.161177 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:07.161182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:07.161185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:07.161216 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:07.164652 (+ 3436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:07.165563 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:07.165568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:07.165574 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:07.165773 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:07.166211 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:07.166216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:07.166220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:07.166472 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:07.166636 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:07.167093 (+ 457us) server_negotiation.cc:299] Negotiation successful
0924 19:03:07.167329 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:03:07.168984 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:07.151161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46223 (local address 127.0.0.1:58750)
0924 19:03:07.151921 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:07.151976 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:07.152142 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:07.153048 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:07.153052 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:07.153681 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:03:07.153923 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:07.153931 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.155372 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.155376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:07.156419 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:07.156434 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.157404 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.157411 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:07.157655 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:07.158331 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:03:07.158364 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:03:07.161001 (+ 2637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:03:07.164783 (+ 3782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:07.164792 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:07.164799 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:07.165394 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:07.165915 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:07.165919 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:07.165922 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:07.166096 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:07.166610 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:03:07.166618 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:03:07.166790 (+ 172us) client_negotiation.cc:769] Sending connection context
0924 19:03:07.166967 (+ 177us) client_negotiation.cc:240] Negotiation successful
0924 19:03:07.168628 (+ 1661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":328,"threads_started":1}
I20260924 19:03:07.170742 14041 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40795 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:03:07 dist-test-slave-9fn1 krb5kdc[13813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 24 19:03:07 dist-test-slave-9fn1 krb5kdc[13813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 19:03:07.184598 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:07.169137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58766 (local address 127.0.97.62:46223)
0924 19:03:07.169295 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:07.169300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:07.169798 (+ 498us) server_negotiation.cc:407] Connection header received
0924 19:03:07.169941 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:07.169948 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:07.170009 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:07.170123 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:07.171671 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.172835 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.174257 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.174962 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.175083 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:07.179434 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:07.179438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:07.179441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:07.179479 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:07.182027 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:07.182692 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:07.182704 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:07.182711 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:07.182826 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:07.183222 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:07.183227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:07.183232 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:07.183492 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:07.183784 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:07.184043 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 19:03:07.184252 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 19:03:07.184934 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:07.171490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:40795 (local address 127.0.0.1:53788)
0924 19:03:07.171658 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:07.171676 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:07.171783 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:07.172246 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:07.172253 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:07.172665 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:03:07.172820 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:07.172829 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.174163 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.174171 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:07.175425 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:07.175436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.176138 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.176145 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:07.176346 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:07.176879 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:03:07.176895 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:03:07.179550 (+ 2655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:03:07.182679 (+ 3129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:07.182686 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:07.182690 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:07.183045 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:07.183418 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:07.183423 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:07.183427 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:07.183564 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:07.184037 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:03:07.184042 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:03:07.184182 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 19:03:07.184317 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 19:03:07.184588 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 19:03:07.184983 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:07.171735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53788 (local address 127.0.97.61:40795)
0924 19:03:07.171931 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:07.171939 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:07.171959 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:03:07.172015 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:07.172020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:07.172097 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:07.172201 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:07.172975 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.174003 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.175564 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.175974 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.176105 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:07.179730 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:07.179735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:07.179739 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:07.179771 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:07.182546 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:07.183164 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:07.183171 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:07.183180 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:07.183304 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:07.183669 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:07.183676 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:07.183679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:07.183924 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:07.184079 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:07.184417 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 19:03:07.184616 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 19:03:07.186013 14041 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:03:07 dist-test-slave-9fn1 krb5kdc[13813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 24 19:03:07 dist-test-slave-9fn1 krb5kdc[13813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 19:03:07.199724 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:07.186816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44917 (local address 127.0.0.1:35178)
0924 19:03:07.186965 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:07.186983 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:07.187068 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:07.187474 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:07.187476 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:07.187790 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:03:07.187909 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:07.187920 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.188977 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.188982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:07.190164 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:07.190176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.190987 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.190993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:07.191186 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:07.191833 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:03:07.191849 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:03:07.193808 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:03:07.197411 (+ 3603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:07.197416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:07.197419 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:07.197812 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:07.198186 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:07.198189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:07.198191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:07.198303 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:07.198834 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:03:07.198842 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:03:07.199005 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 19:03:07.199153 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 19:03:07.199451 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 19:03:07.200042 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:07.187044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35178 (local address 127.0.97.60:44917)
0924 19:03:07.187198 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:07.187203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:07.187221 (+ 18us) server_negotiation.cc:407] Connection header received
0924 19:03:07.187267 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:07.187274 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:07.187345 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:07.187450 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:07.188053 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.188851 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.190298 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.190799 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.190889 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:07.193957 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:07.193961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:07.193964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:07.194012 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:07.197292 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:07.197926 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:07.197932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:07.197936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:07.198069 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:07.198441 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:07.198446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:07.198450 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:07.198714 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:07.198817 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:07.199262 (+ 445us) server_negotiation.cc:299] Negotiation successful
0924 19:03:07.199634 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":40}
I20260924 19:03:07.201069 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:07.186400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53804 (local address 127.0.97.61:40795)
0924 19:03:07.186603 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:07.186608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:07.186626 (+ 18us) server_negotiation.cc:407] Connection header received
0924 19:03:07.186741 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:07.186747 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:07.186814 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:07.186911 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:07.188303 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.189084 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.190461 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.190873 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.190994 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:07.195121 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:07.195128 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:07.195133 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:07.195170 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:07.198283 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:07.198889 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:07.198894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:07.198897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:07.199023 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:07.199417 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:07.199421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:07.199423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:07.199774 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:07.200048 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:07.200511 (+ 463us) server_negotiation.cc:299] Negotiation successful
0924 19:03:07.200733 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 19:03:07.205948 14041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3: Bootstrap starting.
I20260924 19:03:07.206079 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:03:07.210161 14041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3: Neither blocks nor log segments found. Creating new log.
I20260924 19:03:07.211541 14041 log.cc:824] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3: Log is configured to *not* fsync() on all Append() calls
I20260924 19:03:07.221189 14041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3: No bootstrap required, opened a new log
I20260924 19:03:07.233031 14041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537c653c6a848abafdeaa5db27e6b5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46223 } } peers { permanent_uuid: "a6a4ebe2dae640e7bea7465915f5012c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40795 } } peers { permanent_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } }
I20260924 19:03:07.233552 14041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:03:07.233701 14041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b8d5056c9a944d88bdacfb97edbb0b3, State: Initialized, Role: FOLLOWER
I20260924 19:03:07.234409 14041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537c653c6a848abafdeaa5db27e6b5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46223 } } peers { permanent_uuid: "a6a4ebe2dae640e7bea7465915f5012c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40795 } } peers { permanent_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } }
I20260924 19:03:07.236222 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:07.211925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58782 (local address 127.0.97.62:46223)
0924 19:03:07.212434 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:07.212441 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:07.217349 (+ 4908us) server_negotiation.cc:407] Connection header received
0924 19:03:07.217911 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:07.217917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:07.218007 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:07.218084 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:07.220335 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.221381 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.223734 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.224361 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.224467 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:07.228480 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:07.228487 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:07.228492 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:07.228536 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:07.232279 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:07.233651 (+ 1372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:07.233656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:07.233662 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:07.233835 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:07.234463 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:07.234471 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:07.234476 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:07.234802 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:07.235033 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:07.235623 (+ 590us) server_negotiation.cc:299] Negotiation successful
0924 19:03:07.235871 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353}
I20260924 19:03:07.236244 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:07.212494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53806 (local address 127.0.97.61:40795)
0924 19:03:07.216099 (+ 3605us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:07.216107 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:07.216128 (+ 21us) server_negotiation.cc:407] Connection header received
0924 19:03:07.217033 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:07.217040 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:07.217130 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:07.217207 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:07.219918 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.221080 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.223588 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.224220 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.224316 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:07.230061 (+ 5745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:07.230067 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:07.230073 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:07.230121 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:07.233603 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:07.234435 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:07.234440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:07.234444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:07.234575 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:07.234916 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:07.234920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:07.234924 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:07.235198 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:07.235574 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:07.235708 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 19:03:07.235913 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3477}
I20260924 19:03:07.236550 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:07.212169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35184 (local address 127.0.97.60:44917)
0924 19:03:07.212605 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:07.212614 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:07.216756 (+ 4142us) server_negotiation.cc:407] Connection header received
0924 19:03:07.216935 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:07.216950 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:07.217059 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:07.217187 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:07.219935 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.221018 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.223548 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.224271 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.224370 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:07.228289 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:07.228295 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:07.228299 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:07.228338 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:07.232100 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:07.233318 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:07.233324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:07.233329 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:07.233782 (+ 453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:07.234347 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:07.234355 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:07.234360 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:07.234802 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:07.235033 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:07.235989 (+ 956us) server_negotiation.cc:299] Negotiation successful
0924 19:03:07.236206 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294}
I20260924 19:03:07.237439 14054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537c653c6a848abafdeaa5db27e6b5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46223 } } peers { permanent_uuid: "a6a4ebe2dae640e7bea7465915f5012c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40795 } } peers { permanent_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } } }
I20260924 19:03:07.237835 14054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:03:07.238929 14041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:03:07.245736 13974 master_runner.cc:186] Error getting master registration for 127.0.97.62:46223: OK, instance_id { permanent_uuid: "2537c653c6a848abafdeaa5db27e6b5e" instance_seqno: 1790276586159564 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:03:07.253095 13974 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 19:03:07.255539 14065 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:03:07.255673 14065 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:03:07.300379 13893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } attempt: 4
Sep 24 19:03:07 dist-test-slave-9fn1 krb5kdc[13813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260924 19:03:07.315510 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:07.300929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44917 (local address 127.0.0.1:35194)
0924 19:03:07.301158 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:07.301175 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:07.301266 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:07.301619 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:07.301623 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:07.302032 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:03:07.302189 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:07.302198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.303648 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.303652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:07.304538 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:07.304547 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.305486 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.305490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:07.305665 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:07.306280 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:03:07.306297 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:03:07.308450 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:03:07.313110 (+ 4660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:07.313115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:07.313119 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:07.313611 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:07.314005 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:07.314009 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:07.314011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:07.314139 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:07.314780 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:03:07.314783 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:03:07.314906 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 19:03:07.315021 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 19:03:07.315234 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 19:03:07.315918 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:07.301056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35194 (local address 127.0.97.60:44917)
0924 19:03:07.301252 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:07.301259 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:07.301279 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:03:07.301356 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:07.301361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:07.301436 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:07.301581 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:07.302326 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.303521 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.304681 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.305348 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.305486 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:07.308639 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:07.308646 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:07.308651 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:07.308688 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:07.312990 (+ 4302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:07.313717 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:07.313723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:07.313727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:07.313897 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:07.314238 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:07.314244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:07.314248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:07.314631 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:07.314782 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:07.315150 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 19:03:07.315379 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 19:03:07.320194 13893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2537c653c6a848abafdeaa5db27e6b5e: Bootstrap starting.
I20260924 19:03:07.323212 13893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2537c653c6a848abafdeaa5db27e6b5e: Neither blocks nor log segments found. Creating new log.
I20260924 19:03:07.324194 13893 log.cc:824] T 00000000000000000000000000000000 P 2537c653c6a848abafdeaa5db27e6b5e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:03:07.328121 13893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2537c653c6a848abafdeaa5db27e6b5e: No bootstrap required, opened a new log
I20260924 19:03:07.339188 13893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2537c653c6a848abafdeaa5db27e6b5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537c653c6a848abafdeaa5db27e6b5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46223 } } peers { permanent_uuid: "a6a4ebe2dae640e7bea7465915f5012c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40795 } } peers { permanent_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } }
I20260924 19:03:07.339603 13893 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2537c653c6a848abafdeaa5db27e6b5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:03:07.339717 13893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2537c653c6a848abafdeaa5db27e6b5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2537c653c6a848abafdeaa5db27e6b5e, State: Initialized, Role: FOLLOWER
I20260924 19:03:07.340406 13893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2537c653c6a848abafdeaa5db27e6b5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537c653c6a848abafdeaa5db27e6b5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46223 } } peers { permanent_uuid: "a6a4ebe2dae640e7bea7465915f5012c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40795 } } peers { permanent_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } }
I20260924 19:03:07.341434 14068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2537c653c6a848abafdeaa5db27e6b5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537c653c6a848abafdeaa5db27e6b5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46223 } } peers { permanent_uuid: "a6a4ebe2dae640e7bea7465915f5012c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40795 } } peers { permanent_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } } }
I20260924 19:03:07.341849 14068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2537c653c6a848abafdeaa5db27e6b5e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:03:07.342840 13893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2537c653c6a848abafdeaa5db27e6b5e [sys.catalog]: configured and running, proceeding with master startup.
W20260924 19:03:07.361650 14079 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2537c653c6a848abafdeaa5db27e6b5e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:03:07.361819 14079 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 19:03:07.367161 13995 tablet.cc:2369] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:03:07.368592 13967 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } attempt: 5
I20260924 19:03:07.360710 13826 master_runner.cc:186] Error getting master registration for 127.0.97.61:40795: OK, instance_id { permanent_uuid: "a6a4ebe2dae640e7bea7465915f5012c" instance_seqno: 1790276586647562 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:03:07.373471 13826 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 24 19:03:07 dist-test-slave-9fn1 krb5kdc[13813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260924 19:03:07.387233 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:07.369193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:44917 (local address 127.0.0.1:35210)
0924 19:03:07.369380 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:07.369404 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:07.369547 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:07.370146 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:07.370151 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:07.370648 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:03:07.370874 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:07.370888 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.372455 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.372460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:07.373953 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:07.373968 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.376512 (+ 2544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.376518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:07.376713 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:07.377409 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:03:07.377433 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:03:07.380236 (+ 2803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:03:07.384472 (+ 4236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:07.384477 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:07.384481 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:07.384976 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:07.385393 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:07.385398 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:07.385402 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:07.385582 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:07.386222 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:03:07.386227 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:03:07.386380 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 19:03:07.386550 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 19:03:07.386825 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 19:03:07.387418 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:07.369566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35210 (local address 127.0.97.60:44917)
0924 19:03:07.369788 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:07.369794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:07.369816 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:03:07.369874 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:07.369880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:07.369966 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:07.370085 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:07.371067 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.372252 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.374104 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:07.374779 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:07.374903 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:07.380425 (+ 5522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:07.380437 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:07.380444 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:07.380488 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:07.384296 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:07.385117 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:07.385123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:07.385127 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:07.385259 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:07.385728 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:07.385735 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:07.385739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:07.386087 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:07.386261 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:07.386774 (+ 513us) server_negotiation.cc:299] Negotiation successful
0924 19:03:07.387021 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260924 19:03:07.393461 13967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6a4ebe2dae640e7bea7465915f5012c: Bootstrap starting.
I20260924 19:03:07.398533 13967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6a4ebe2dae640e7bea7465915f5012c: Neither blocks nor log segments found. Creating new log.
W20260924 19:03:07.399253 13847 tablet.cc:2369] T 00000000000000000000000000000000 P 2537c653c6a848abafdeaa5db27e6b5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:03:07.400794 13967 log.cc:824] T 00000000000000000000000000000000 P a6a4ebe2dae640e7bea7465915f5012c: Log is configured to *not* fsync() on all Append() calls
I20260924 19:03:07.404174 13967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6a4ebe2dae640e7bea7465915f5012c: No bootstrap required, opened a new log
I20260924 19:03:07.415180 13967 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6a4ebe2dae640e7bea7465915f5012c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537c653c6a848abafdeaa5db27e6b5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46223 } } peers { permanent_uuid: "a6a4ebe2dae640e7bea7465915f5012c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40795 } } peers { permanent_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } }
I20260924 19:03:07.415573 13967 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6a4ebe2dae640e7bea7465915f5012c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:03:07.415680 13967 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6a4ebe2dae640e7bea7465915f5012c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a4ebe2dae640e7bea7465915f5012c, State: Initialized, Role: FOLLOWER
I20260924 19:03:07.416409 13967 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6a4ebe2dae640e7bea7465915f5012c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537c653c6a848abafdeaa5db27e6b5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46223 } } peers { permanent_uuid: "a6a4ebe2dae640e7bea7465915f5012c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40795 } } peers { permanent_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } }
I20260924 19:03:07.418327 14082 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6a4ebe2dae640e7bea7465915f5012c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537c653c6a848abafdeaa5db27e6b5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46223 } } peers { permanent_uuid: "a6a4ebe2dae640e7bea7465915f5012c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40795 } } peers { permanent_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } } }
I20260924 19:03:07.418813 14082 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6a4ebe2dae640e7bea7465915f5012c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:03:07.419656 13967 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6a4ebe2dae640e7bea7465915f5012c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:03:07.433753 13900 master_runner.cc:263] No leader master found from master a6a4ebe2dae640e7bea7465915f5012c
I20260924 19:03:07.433915 13900 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 19:03:07.440198 14093 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a6a4ebe2dae640e7bea7465915f5012c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:03:07.440418 14093 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:03:07.493186 14054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:03:07.493564 14054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537c653c6a848abafdeaa5db27e6b5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46223 } } peers { permanent_uuid: "a6a4ebe2dae640e7bea7465915f5012c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40795 } } peers { permanent_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } }
I20260924 19:03:07.495332 14054 leader_election.cc:288] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2537c653c6a848abafdeaa5db27e6b5e (127.0.97.62:46223), a6a4ebe2dae640e7bea7465915f5012c (127.0.97.61:40795)
I20260924 19:03:07.496596 13942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a4ebe2dae640e7bea7465915f5012c" is_pre_election: true
I20260924 19:03:07.497236 13942 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a6a4ebe2dae640e7bea7465915f5012c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b8d5056c9a944d88bdacfb97edbb0b3 in term 0.
I20260924 19:03:07.498016 13990 leader_election.cc:302] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b8d5056c9a944d88bdacfb97edbb0b3, a6a4ebe2dae640e7bea7465915f5012c; no voters:
I20260924 19:03:07.499269 13868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2537c653c6a848abafdeaa5db27e6b5e" is_pre_election: true
I20260924 19:03:07.499857 13868 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2537c653c6a848abafdeaa5db27e6b5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b8d5056c9a944d88bdacfb97edbb0b3 in term 0.
I20260924 19:03:07.500864 14054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:03:07.501019 14054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:03:07.501122 14054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:03:07.504623 14054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537c653c6a848abafdeaa5db27e6b5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46223 } } peers { permanent_uuid: "a6a4ebe2dae640e7bea7465915f5012c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40795 } } peers { permanent_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } }
I20260924 19:03:07.505707 13868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2537c653c6a848abafdeaa5db27e6b5e"
I20260924 19:03:07.505926 13868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2537c653c6a848abafdeaa5db27e6b5e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:03:07.509755 13868 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2537c653c6a848abafdeaa5db27e6b5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b8d5056c9a944d88bdacfb97edbb0b3 in term 1.
I20260924 19:03:07.510279 13993 leader_election.cc:302] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2537c653c6a848abafdeaa5db27e6b5e, 6b8d5056c9a944d88bdacfb97edbb0b3; no voters:
I20260924 19:03:07.518000 13942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6a4ebe2dae640e7bea7465915f5012c"
I20260924 19:03:07.518245 13942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6a4ebe2dae640e7bea7465915f5012c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:03:07.522347 13942 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a6a4ebe2dae640e7bea7465915f5012c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b8d5056c9a944d88bdacfb97edbb0b3 in term 1.
I20260924 19:03:07.523247 14054 leader_election.cc:288] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3 [CANDIDATE]: Term 1 election: Requested vote from peers 2537c653c6a848abafdeaa5db27e6b5e (127.0.97.62:46223), a6a4ebe2dae640e7bea7465915f5012c (127.0.97.61:40795)
I20260924 19:03:07.523663 14054 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:03:07.524083 14054 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3 [term 1 LEADER]: Becoming Leader. State: Replica: 6b8d5056c9a944d88bdacfb97edbb0b3, State: Running, Role: LEADER
I20260924 19:03:07.524923 14054 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537c653c6a848abafdeaa5db27e6b5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46223 } } peers { permanent_uuid: "a6a4ebe2dae640e7bea7465915f5012c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40795 } } peers { permanent_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } }
I20260924 19:03:07.538754 14095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b8d5056c9a944d88bdacfb97edbb0b3. Latest consensus state: current_term: 1 leader_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537c653c6a848abafdeaa5db27e6b5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46223 } } peers { permanent_uuid: "a6a4ebe2dae640e7bea7465915f5012c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40795 } } peers { permanent_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } } }
I20260924 19:03:07.539382 14095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3 [sys.catalog]: This master's current role is: LEADER
I20260924 19:03:07.552347 14097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:03:07.554876 14097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:03:07.580166 13868 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2537c653c6a848abafdeaa5db27e6b5e [term 1 FOLLOWER]: Refusing update from remote peer 6b8d5056c9a944d88bdacfb97edbb0b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:03:07.581374 14054 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2537c653c6a848abafdeaa5db27e6b5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:03:07.582614 13942 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a6a4ebe2dae640e7bea7465915f5012c [term 1 FOLLOWER]: Refusing update from remote peer 6b8d5056c9a944d88bdacfb97edbb0b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:03:07.583947 14054 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6a4ebe2dae640e7bea7465915f5012c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:03:07.603747 14068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2537c653c6a848abafdeaa5db27e6b5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b8d5056c9a944d88bdacfb97edbb0b3. Latest consensus state: current_term: 1 leader_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537c653c6a848abafdeaa5db27e6b5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46223 } } peers { permanent_uuid: "a6a4ebe2dae640e7bea7465915f5012c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40795 } } peers { permanent_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } } }
I20260924 19:03:07.604066 14068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2537c653c6a848abafdeaa5db27e6b5e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:03:07.606390 14098 mvcc.cc:205] Tried to move back new op lower bound from 7332972902711586816 to 7332972902515884032. Current Snapshot: MvccSnapshot[applied={T|T < 7332972902711586816}]
I20260924 19:03:07.609997 14082 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6a4ebe2dae640e7bea7465915f5012c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b8d5056c9a944d88bdacfb97edbb0b3. Latest consensus state: current_term: 1 leader_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537c653c6a848abafdeaa5db27e6b5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46223 } } peers { permanent_uuid: "a6a4ebe2dae640e7bea7465915f5012c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40795 } } peers { permanent_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } } }
I20260924 19:03:07.610296 14082 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6a4ebe2dae640e7bea7465915f5012c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:03:07.626360 14101 mvcc.cc:205] Tried to move back new op lower bound from 7332972902711586816 to 7332972902515884032. Current Snapshot: MvccSnapshot[applied={T|T < 7332972902711586816}]
I20260924 19:03:07.626756 14054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537c653c6a848abafdeaa5db27e6b5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46223 } } peers { permanent_uuid: "a6a4ebe2dae640e7bea7465915f5012c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40795 } } peers { permanent_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } } }
I20260924 19:03:07.627101 14054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3 [sys.catalog]: This master's current role is: LEADER
I20260924 19:03:07.627424 14054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537c653c6a848abafdeaa5db27e6b5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46223 } } peers { permanent_uuid: "a6a4ebe2dae640e7bea7465915f5012c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40795 } } peers { permanent_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } } }
I20260924 19:03:07.627831 14054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3 [sys.catalog]: This master's current role is: LEADER
I20260924 19:03:07.631126 14097 catalog_manager.cc:1347] Generated new cluster ID: 308ba7f329f54951944eee536eaafcb2
I20260924 19:03:07.631253 14097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:03:07.633838 14082 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6a4ebe2dae640e7bea7465915f5012c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537c653c6a848abafdeaa5db27e6b5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46223 } } peers { permanent_uuid: "a6a4ebe2dae640e7bea7465915f5012c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40795 } } peers { permanent_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } } }
I20260924 19:03:07.634414 14082 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6a4ebe2dae640e7bea7465915f5012c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:03:07.651762 14068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2537c653c6a848abafdeaa5db27e6b5e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2537c653c6a848abafdeaa5db27e6b5e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46223 } } peers { permanent_uuid: "a6a4ebe2dae640e7bea7465915f5012c" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 40795 } } peers { permanent_uuid: "6b8d5056c9a944d88bdacfb97edbb0b3" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 44917 } } }
I20260924 19:03:07.652025 14068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2537c653c6a848abafdeaa5db27e6b5e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:03:07.671823 14097 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:03:07.673848 14097 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:03:07.715508 14097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b8d5056c9a944d88bdacfb97edbb0b3: Generated new TSK 0
I20260924 19:03:07.716632 14097 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:03:08.364418 14079 catalog_manager.cc:1259] Loaded cluster ID: 308ba7f329f54951944eee536eaafcb2
I20260924 19:03:08.364544 14079 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2537c653c6a848abafdeaa5db27e6b5e: loading cluster ID for follower catalog manager: success
I20260924 19:03:08.367100 14079 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2537c653c6a848abafdeaa5db27e6b5e: acquiring CA information for follower catalog manager: success
I20260924 19:03:08.368849 14079 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2537c653c6a848abafdeaa5db27e6b5e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:03:08.442652 14093 catalog_manager.cc:1259] Loaded cluster ID: 308ba7f329f54951944eee536eaafcb2
I20260924 19:03:08.442764 14093 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a6a4ebe2dae640e7bea7465915f5012c: loading cluster ID for follower catalog manager: success
I20260924 19:03:08.445279 14093 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a6a4ebe2dae640e7bea7465915f5012c: acquiring CA information for follower catalog manager: success
I20260924 19:03:08.447016 14093 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a6a4ebe2dae640e7bea7465915f5012c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:03:08.821204 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 13826
I20260924 19:03:08.822460 13888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:03:08.933943 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 13826
I20260924 19:03:08.955233 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 13900
I20260924 19:03:08.956408 13962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:03:09.061022 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 13900
I20260924 19:03:09.081014 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 13974
I20260924 19:03:09.082149 14036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:03:09.189589 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 13974
2026-09-24T19:03:09Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4764 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:03:09 dist-test-slave-9fn1 krb5kdc[14116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:03:09 dist-test-slave-9fn1 krb5kdc[14116](info): set up 2 sockets
Sep 24 19:03:09 dist-test-slave-9fn1 krb5kdc[14116](info): commencing operation
krb5kdc: starting...
W20260924 19:03:10.273914 388 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:03:10 dist-test-slave-9fn1 krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:03:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:03:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 19:03:10.513823 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40153
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40153
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:03:10.841831 14129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:03:10.842185 14129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:03:10.842357 14129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:03:10.849890 14129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:03:10.850035 14129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:03:10.850149 14129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:03:10.850252 14129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:03:10.850337 14129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:03:10.860733 14129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:03:10.862169 14129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:03:10.863502 14129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:03:10.874929 14134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:10.874963 14135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:10.875000 14137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:03:10.875767 14129 server_base.cc:1034] running on GCE node
I20260924 19:03:10.876588 14129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:03:10.878036 14129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:03:10.879283 14129 hybrid_clock.cc:648] HybridClock initialized: now 1790276590879247 us; error 43 us; skew 500 ppm
I20260924 19:03:10.879671 14129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:03:10 dist-test-slave-9fn1 krb5kdc[14116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:03:10.882786 14129 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:03:10.884379 14129 webserver.cc:460] Webserver started at http://127.0.97.62:41303/ using document root <none> and password file <none>
I20260924 19:03:10.885089 14129 fs_manager.cc:359] Metadata directory not provided
I20260924 19:03:10.885238 14129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:03:10.885643 14129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:03:10.888568 14129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "5ccb5fbf84e2454c8e12db72ed88ed55"
format_stamp: "Formatted at 2026-09-24 19:03:10 on dist-test-slave-9fn1"
server_key: "9624965a099f153905b22b1cdc146eec"
server_key_iv: "343c1abf9bbe8b30c3c914450c14a129"
server_key_version: "encryptionkey@0"
I20260924 19:03:10.889310 14129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "5ccb5fbf84e2454c8e12db72ed88ed55"
format_stamp: "Formatted at 2026-09-24 19:03:10 on dist-test-slave-9fn1"
server_key: "9624965a099f153905b22b1cdc146eec"
server_key_iv: "343c1abf9bbe8b30c3c914450c14a129"
server_key_version: "encryptionkey@0"
I20260924 19:03:10.894488 14129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 19:03:10.897733 14144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:10.898998 14129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:03:10.899202 14129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "5ccb5fbf84e2454c8e12db72ed88ed55"
format_stamp: "Formatted at 2026-09-24 19:03:10 on dist-test-slave-9fn1"
server_key: "9624965a099f153905b22b1cdc146eec"
server_key_iv: "343c1abf9bbe8b30c3c914450c14a129"
server_key_version: "encryptionkey@0"
I20260924 19:03:10.899407 14129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:03:10.915418 14129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:03:10.919315 14129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:03:10.919593 14129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:03:10.937748 14129 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40153
I20260924 19:03:10.937772 14195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40153 every 8 connection(s)
I20260924 19:03:10.939059 14129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:03:10.942719 14196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:03:10.945065 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 14129
I20260924 19:03:10.945288 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:03:10.945948 388 external_mini_cluster.cc:1428] Setting key bc0ebc7023b53f132f980136f63e44c6
I20260924 19:03:10.955338 14196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ccb5fbf84e2454c8e12db72ed88ed55: Bootstrap starting.
I20260924 19:03:10.958794 14196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ccb5fbf84e2454c8e12db72ed88ed55: Neither blocks nor log segments found. Creating new log.
I20260924 19:03:10.959733 14196 log.cc:824] T 00000000000000000000000000000000 P 5ccb5fbf84e2454c8e12db72ed88ed55: Log is configured to *not* fsync() on all Append() calls
Sep 24 19:03:10 dist-test-slave-9fn1 krb5kdc[14116](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:03:10.962803 14196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ccb5fbf84e2454c8e12db72ed88ed55: No bootstrap required, opened a new log
I20260924 19:03:10.969318 14199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:10.947745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35450 (local address 127.0.97.62:40153)
0924 19:03:10.948544 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:10.948560 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:10.948607 (+ 47us) server_negotiation.cc:407] Connection header received
0924 19:03:10.950964 (+ 2357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:10.950993 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:10.951509 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:10.951953 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:10.953240 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:10.955159 (+ 1919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:10.956488 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:10.957593 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:10.957765 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:10.962182 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:10.962189 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:10.962203 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:10.962236 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:10.965282 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:10.966147 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:10.966153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:10.966158 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:10.966339 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:10.966757 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:10.966760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:10.966761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:10.967252 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:10.967488 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:10.967943 (+ 455us) server_negotiation.cc:299] Negotiation successful
0924 19:03:10.968308 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":322,"threads_started":1}
I20260924 19:03:10.971422 14196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ccb5fbf84e2454c8e12db72ed88ed55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ccb5fbf84e2454c8e12db72ed88ed55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40153 } }
I20260924 19:03:10.971832 14196 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ccb5fbf84e2454c8e12db72ed88ed55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:03:10.971956 14196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ccb5fbf84e2454c8e12db72ed88ed55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ccb5fbf84e2454c8e12db72ed88ed55, State: Initialized, Role: FOLLOWER
I20260924 19:03:10.972579 14196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ccb5fbf84e2454c8e12db72ed88ed55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ccb5fbf84e2454c8e12db72ed88ed55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40153 } }
I20260924 19:03:10.972815 14196 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ccb5fbf84e2454c8e12db72ed88ed55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:03:10.972983 14196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ccb5fbf84e2454c8e12db72ed88ed55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:03:10.973141 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:03:10.973168 14196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ccb5fbf84e2454c8e12db72ed88ed55 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:03:10.975329 14196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ccb5fbf84e2454c8e12db72ed88ed55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ccb5fbf84e2454c8e12db72ed88ed55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40153 } }
I20260924 19:03:10.975840 14196 leader_election.cc:302] T 00000000000000000000000000000000 P 5ccb5fbf84e2454c8e12db72ed88ed55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ccb5fbf84e2454c8e12db72ed88ed55; no voters:
I20260924 19:03:10.976445 14196 leader_election.cc:288] T 00000000000000000000000000000000 P 5ccb5fbf84e2454c8e12db72ed88ed55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:03:10.976531 14201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ccb5fbf84e2454c8e12db72ed88ed55 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:03:10.977231 14201 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ccb5fbf84e2454c8e12db72ed88ed55 [term 1 LEADER]: Becoming Leader. State: Replica: 5ccb5fbf84e2454c8e12db72ed88ed55, State: Running, Role: LEADER
I20260924 19:03:10.977802 14201 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ccb5fbf84e2454c8e12db72ed88ed55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ccb5fbf84e2454c8e12db72ed88ed55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40153 } }
I20260924 19:03:10.978376 14196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ccb5fbf84e2454c8e12db72ed88ed55 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:03:10.980818 14203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ccb5fbf84e2454c8e12db72ed88ed55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ccb5fbf84e2454c8e12db72ed88ed55. Latest consensus state: current_term: 1 leader_uuid: "5ccb5fbf84e2454c8e12db72ed88ed55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ccb5fbf84e2454c8e12db72ed88ed55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40153 } } }
I20260924 19:03:10.980849 14202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ccb5fbf84e2454c8e12db72ed88ed55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ccb5fbf84e2454c8e12db72ed88ed55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ccb5fbf84e2454c8e12db72ed88ed55" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40153 } } }
I20260924 19:03:10.981184 14203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ccb5fbf84e2454c8e12db72ed88ed55 [sys.catalog]: This master's current role is: LEADER
I20260924 19:03:10.981210 14202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ccb5fbf84e2454c8e12db72ed88ed55 [sys.catalog]: This master's current role is: LEADER
I20260924 19:03:10.986402 14207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:03:10.990342 14207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:03:10.997608 14207 catalog_manager.cc:1347] Generated new cluster ID: a268eda9f1894ecd823e8faab1d25756
I20260924 19:03:10.997711 14207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:03:11.025137 14207 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:03:11.027097 14207 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:03:11.040977 14207 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ccb5fbf84e2454c8e12db72ed88ed55: Generated new TSK 0
I20260924 19:03:11.041899 14207 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:03:11.085278 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 14129
I20260924 19:03:11.086477 14191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:03:11.182828 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 14129
2026-09-24T19:03:11Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1991 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:03:11 dist-test-slave-9fn1 krb5kdc[14224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:03:11 dist-test-slave-9fn1 krb5kdc[14224](info): set up 2 sockets
Sep 24 19:03:11 dist-test-slave-9fn1 krb5kdc[14224](info): commencing operation
krb5kdc: starting...
W20260924 19:03:12.269817 388 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:03:12 dist-test-slave-9fn1 krb5kdc[14224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:03:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:03:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 19:03:12.505038 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45755
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:03:12.826774 14237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:03:12.827054 14237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:03:12.827226 14237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:03:12.834399 14237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:03:12.834497 14237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:03:12.834611 14237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:03:12.834719 14237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:03:12.834806 14237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:03:12.844480 14237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:03:12.845798 14237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:03:12.847123 14237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:03:12.858678 14243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:12.858800 14242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:12.858778 14245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:03:12.859902 14237 server_base.cc:1034] running on GCE node
I20260924 19:03:12.860711 14237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:03:12.862147 14237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:03:12.863358 14237 hybrid_clock.cc:648] HybridClock initialized: now 1790276592863309 us; error 42 us; skew 500 ppm
I20260924 19:03:12.863727 14237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:03:12 dist-test-slave-9fn1 krb5kdc[14224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276592, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:03:12.875459 14237 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:03:12.876938 14237 webserver.cc:460] Webserver started at http://127.0.97.62:35637/ using document root <none> and password file <none>
I20260924 19:03:12.877697 14237 fs_manager.cc:359] Metadata directory not provided
I20260924 19:03:12.877846 14237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:03:12.878144 14237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:03:12.881093 14237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "8e301c22a72249bc9d55d0eda728c70b"
format_stamp: "Formatted at 2026-09-24 19:03:12 on dist-test-slave-9fn1"
server_key: "23db884e266ad7796146c591501f6a3d"
server_key_iv: "8a931e9f2fff36850c1a5ad7ee20c339"
server_key_version: "encryptionkey@0"
I20260924 19:03:12.881928 14237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "8e301c22a72249bc9d55d0eda728c70b"
format_stamp: "Formatted at 2026-09-24 19:03:12 on dist-test-slave-9fn1"
server_key: "23db884e266ad7796146c591501f6a3d"
server_key_iv: "8a931e9f2fff36850c1a5ad7ee20c339"
server_key_version: "encryptionkey@0"
I20260924 19:03:12.886997 14237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 19:03:12.890347 14252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:12.891630 14237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:03:12.891819 14237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "8e301c22a72249bc9d55d0eda728c70b"
format_stamp: "Formatted at 2026-09-24 19:03:12 on dist-test-slave-9fn1"
server_key: "23db884e266ad7796146c591501f6a3d"
server_key_iv: "8a931e9f2fff36850c1a5ad7ee20c339"
server_key_version: "encryptionkey@0"
I20260924 19:03:12.892027 14237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:03:12.917961 14237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:03:12.922312 14237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:03:12.922621 14237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:03:12.940626 14237 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45755
I20260924 19:03:12.940655 14303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45755 every 8 connection(s)
I20260924 19:03:12.942039 14237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:03:12.945698 14304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:03:12.946314 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 14237
I20260924 19:03:12.946544 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:03:12.947409 388 external_mini_cluster.cc:1428] Setting key 09f1a2640c40fd534b6cefbb7a354017
I20260924 19:03:12.959820 14304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b: Bootstrap starting.
Sep 24 19:03:12 dist-test-slave-9fn1 krb5kdc[14224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:03:12.962976 14304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b: Neither blocks nor log segments found. Creating new log.
I20260924 19:03:12.964077 14304 log.cc:824] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b: Log is configured to *not* fsync() on all Append() calls
I20260924 19:03:12.967809 14304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b: No bootstrap required, opened a new log
I20260924 19:03:12.973109 14307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:12.949609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49616 (local address 127.0.97.62:45755)
0924 19:03:12.950377 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:12.950392 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:12.950454 (+ 62us) server_negotiation.cc:407] Connection header received
0924 19:03:12.952247 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:12.952274 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:12.952756 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:12.953180 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:12.954683 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:12.956831 (+ 2148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:12.958316 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:12.959321 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:12.959520 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:12.964065 (+ 4545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:12.964078 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:12.964098 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:12.964145 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:12.968318 (+ 4173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:12.969123 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:12.969137 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:12.969145 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:12.969367 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:12.969832 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:12.969837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:12.969841 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:12.970568 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:12.970871 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:12.971297 (+ 426us) server_negotiation.cc:299] Negotiation successful
0924 19:03:12.971762 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":543,"thread_start_us":330,"threads_started":1}
I20260924 19:03:12.977605 14304 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e301c22a72249bc9d55d0eda728c70b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45755 } }
I20260924 19:03:12.977937 14304 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:03:12.978020 14304 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e301c22a72249bc9d55d0eda728c70b, State: Initialized, Role: FOLLOWER
I20260924 19:03:12.978559 14304 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e301c22a72249bc9d55d0eda728c70b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45755 } }
I20260924 19:03:12.978758 14304 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:03:12.978852 14304 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:03:12.978991 14304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:03:12.981122 14304 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e301c22a72249bc9d55d0eda728c70b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45755 } }
I20260924 19:03:12.981650 14304 leader_election.cc:302] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e301c22a72249bc9d55d0eda728c70b; no voters:
I20260924 19:03:12.982223 14304 leader_election.cc:288] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:03:12.982396 14309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:03:12.983114 14309 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [term 1 LEADER]: Becoming Leader. State: Replica: 8e301c22a72249bc9d55d0eda728c70b, State: Running, Role: LEADER
I20260924 19:03:12.983660 14309 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e301c22a72249bc9d55d0eda728c70b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45755 } }
I20260924 19:03:12.984308 14304 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:03:12.986231 14310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e301c22a72249bc9d55d0eda728c70b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e301c22a72249bc9d55d0eda728c70b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45755 } } }
I20260924 19:03:12.986304 14311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e301c22a72249bc9d55d0eda728c70b. Latest consensus state: current_term: 1 leader_uuid: "8e301c22a72249bc9d55d0eda728c70b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e301c22a72249bc9d55d0eda728c70b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45755 } } }
I20260924 19:03:12.986590 14311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [sys.catalog]: This master's current role is: LEADER
I20260924 19:03:12.986590 14310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [sys.catalog]: This master's current role is: LEADER
I20260924 19:03:12.987571 14316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:03:12.993184 14316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:03:13.001879 14316 catalog_manager.cc:1347] Generated new cluster ID: ab8033854edd473da4e1ed318474d999
I20260924 19:03:13.001992 14316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:03:13.025511 14316 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:03:13.026885 14316 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:03:13.036514 14316 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b: Generated new TSK 0
I20260924 19:03:13.037307 14316 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 19:03:13.101464 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45755
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:03:13.443179 14330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:03:13.443434 14330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:03:13.443523 14330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:03:13.450631 14330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:03:13.450737 14330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:03:13.450847 14330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:03:13.460457 14330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:03:13.461683 14330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:03:13.462888 14330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:03:13.475396 14335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:13.475500 14336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:13.475808 14338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:03:13.476552 14330 server_base.cc:1034] running on GCE node
I20260924 19:03:13.477108 14330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:03:13.478044 14330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:03:13.479264 14330 hybrid_clock.cc:648] HybridClock initialized: now 1790276593479235 us; error 43 us; skew 500 ppm
I20260924 19:03:13.479559 14330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:03:13 dist-test-slave-9fn1 krb5kdc[14224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:03:13.482527 14330 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 19:03:13.483819 14330 webserver.cc:460] Webserver started at http://127.0.97.1:33025/ using document root <none> and password file <none>
I20260924 19:03:13.484534 14330 fs_manager.cc:359] Metadata directory not provided
I20260924 19:03:13.484683 14330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:03:13.484969 14330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:03:13.487913 14330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/data/instance:
uuid: "fc0a539cb7784d15a1150b61ad525ce8"
format_stamp: "Formatted at 2026-09-24 19:03:13 on dist-test-slave-9fn1"
server_key: "81deae84fae9525c71a42bb55270b7ce"
server_key_iv: "0e0de2429461d95bf0dab6a086a14d59"
server_key_version: "encryptionkey@0"
I20260924 19:03:13.488660 14330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/wal/instance:
uuid: "fc0a539cb7784d15a1150b61ad525ce8"
format_stamp: "Formatted at 2026-09-24 19:03:13 on dist-test-slave-9fn1"
server_key: "81deae84fae9525c71a42bb55270b7ce"
server_key_iv: "0e0de2429461d95bf0dab6a086a14d59"
server_key_version: "encryptionkey@0"
I20260924 19:03:13.493419 14330 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260924 19:03:13.496845 14345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:13.498099 14330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 19:03:13.498286 14330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "fc0a539cb7784d15a1150b61ad525ce8"
format_stamp: "Formatted at 2026-09-24 19:03:13 on dist-test-slave-9fn1"
server_key: "81deae84fae9525c71a42bb55270b7ce"
server_key_iv: "0e0de2429461d95bf0dab6a086a14d59"
server_key_version: "encryptionkey@0"
I20260924 19:03:13.498492 14330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:03:13.514885 14330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:03:13.519084 14330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:03:13.519411 14330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:03:13.520422 14330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:03:13.521898 14330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:03:13.522007 14330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:13.522176 14330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:03:13.522257 14330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:13.557299 14330 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39497
I20260924 19:03:13.557354 14457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39497 every 8 connection(s)
I20260924 19:03:13.558633 14330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:03:13.567443 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 14330
I20260924 19:03:13.567653 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/wal/instance
I20260924 19:03:13.568327 388 external_mini_cluster.cc:1428] Setting key abf484aed0c378765b8e019f785a9de4
Sep 24 19:03:13 dist-test-slave-9fn1 krb5kdc[14224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:03:13.580396 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:13.561164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55905 (local address 127.0.97.62:45755)
0924 19:03:13.561629 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:13.561634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:13.562570 (+ 936us) server_negotiation.cc:407] Connection header received
0924 19:03:13.564334 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:13.564340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:13.564439 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:13.564557 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:13.566032 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:13.566745 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:13.568501 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:13.568929 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:13.569124 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:13.573048 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:13.573058 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:13.573066 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:13.573113 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:13.576908 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:13.577804 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:13.577812 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:13.577819 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:13.577981 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:13.578422 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:13.578427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:13.578429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:13.578715 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:13.578880 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:13.579729 (+ 849us) server_negotiation.cc:299] Negotiation successful
0924 19:03:13.579973 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":233,"threads_started":1}
I20260924 19:03:13.581461 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:13.561711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.1:55905)
0924 19:03:13.562404 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:13.562448 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:13.564070 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:13.564787 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:13.564795 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:13.565278 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:03:13.565856 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:13.565872 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:13.566904 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:13.566912 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:13.568306 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:13.568318 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:13.569289 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:13.569298 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:13.569617 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:13.570367 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:03:13.570395 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:03:13.572809 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:03:13.577069 (+ 4260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:13.577077 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:13.577090 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:13.577656 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:13.578091 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:13.578095 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:13.578098 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:13.578278 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:13.578862 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:03:13.578870 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:03:13.579228 (+ 358us) client_negotiation.cc:769] Sending connection context
0924 19:03:13.579606 (+ 378us) client_negotiation.cc:240] Negotiation successful
0924 19:03:13.580189 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":268,"threads_started":1}
I20260924 19:03:13.583163 14458 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45755
I20260924 19:03:13.583477 14458 heartbeater.cc:464] Registering TS with master...
I20260924 19:03:13.584327 14458 heartbeater.cc:511] Master 127.0.97.62:45755 requested a full tablet report, sending...
I20260924 19:03:13.586673 14269 ts_manager.cc:194] Registered new tserver with Master: fc0a539cb7784d15a1150b61ad525ce8 (127.0.97.1:39497)
I20260924 19:03:13.588968 14269 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55905
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 19:03:13.622906 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45755
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:03:13.950996 14464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:03:13.951334 14464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:03:13.951478 14464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:03:13.960935 14464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:03:13.961097 14464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:03:13.961277 14464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:03:13.970535 14464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:03:13.971793 14464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:03:13.973032 14464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:03:13.985769 14469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:13.985760 14472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:13.985725 14470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:03:13.986708 14464 server_base.cc:1034] running on GCE node
I20260924 19:03:13.987211 14464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:03:13.988116 14464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:03:13.989348 14464 hybrid_clock.cc:648] HybridClock initialized: now 1790276593989293 us; error 62 us; skew 500 ppm
I20260924 19:03:13.989696 14464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:03:13 dist-test-slave-9fn1 krb5kdc[14224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:03:13.992621 14464 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 19:03:13.994095 14464 webserver.cc:460] Webserver started at http://127.0.97.2:33037/ using document root <none> and password file <none>
I20260924 19:03:13.994796 14464 fs_manager.cc:359] Metadata directory not provided
I20260924 19:03:13.994894 14464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:03:13.995169 14464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:03:13.998136 14464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/data/instance:
uuid: "c55cf3a0245f4282a606a576da7f8c6d"
format_stamp: "Formatted at 2026-09-24 19:03:13 on dist-test-slave-9fn1"
server_key: "c65d999ab6ce624601c38519969218b5"
server_key_iv: "85a4ddea9b043f6b1820caec9b8aace5"
server_key_version: "encryptionkey@0"
I20260924 19:03:13.998893 14464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c55cf3a0245f4282a606a576da7f8c6d"
format_stamp: "Formatted at 2026-09-24 19:03:13 on dist-test-slave-9fn1"
server_key: "c65d999ab6ce624601c38519969218b5"
server_key_iv: "85a4ddea9b043f6b1820caec9b8aace5"
server_key_version: "encryptionkey@0"
I20260924 19:03:14.003860 14464 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 19:03:14.007306 14479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:14.008635 14464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:03:14.008790 14464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "c55cf3a0245f4282a606a576da7f8c6d"
format_stamp: "Formatted at 2026-09-24 19:03:13 on dist-test-slave-9fn1"
server_key: "c65d999ab6ce624601c38519969218b5"
server_key_iv: "85a4ddea9b043f6b1820caec9b8aace5"
server_key_version: "encryptionkey@0"
I20260924 19:03:14.008973 14464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:03:14.033897 14464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:03:14.037719 14464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:03:14.038012 14464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:03:14.038914 14464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:03:14.040210 14464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:03:14.040280 14464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:14.040433 14464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:03:14.040493 14464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:14.073133 14464 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40787
I20260924 19:03:14.073230 14591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40787 every 8 connection(s)
I20260924 19:03:14.074450 14464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:03:14.083053 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 14464
I20260924 19:03:14.083254 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/wal/instance
I20260924 19:03:14.083678 388 external_mini_cluster.cc:1428] Setting key ec77b3b09ce4486c2be9af33bcb8329f
Sep 24 19:03:14 dist-test-slave-9fn1 krb5kdc[14224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:03:14.095588 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:14.077058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44707 (local address 127.0.97.62:45755)
0924 19:03:14.077232 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:14.077236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:14.078324 (+ 1088us) server_negotiation.cc:407] Connection header received
0924 19:03:14.080125 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:14.080130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:14.080206 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:14.080297 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:14.081709 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.082448 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.084050 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.084684 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.084808 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:14.088315 (+ 3507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:14.088324 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:14.088332 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:14.088376 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:14.092257 (+ 3881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:14.092972 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:14.092980 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:14.092986 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:14.093153 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:14.093593 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:14.093598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:14.093603 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:14.093933 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:14.094044 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:14.095032 (+ 988us) server_negotiation.cc:299] Negotiation successful
0924 19:03:14.095257 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 19:03:14.096937 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:14.077435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.2:44707)
0924 19:03:14.078159 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:14.078196 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:14.079828 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:14.080509 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:14.080517 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:14.080994 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:03:14.081554 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:14.081571 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.082581 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.082584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:14.083916 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:14.083933 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.084931 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.084938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:14.085220 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:14.085952 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:03:14.085982 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:03:14.088089 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:03:14.092410 (+ 4321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:14.092417 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:14.092429 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:14.092836 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:14.093258 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:14.093261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:14.093263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:14.093442 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:14.094065 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:03:14.094076 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:03:14.094556 (+ 480us) client_negotiation.cc:769] Sending connection context
0924 19:03:14.094887 (+ 331us) client_negotiation.cc:240] Negotiation successful
0924 19:03:14.095479 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":261,"threads_started":1}
I20260924 19:03:14.098511 14592 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45755
I20260924 19:03:14.098817 14592 heartbeater.cc:464] Registering TS with master...
I20260924 19:03:14.099500 14592 heartbeater.cc:511] Master 127.0.97.62:45755 requested a full tablet report, sending...
I20260924 19:03:14.100978 14269 ts_manager.cc:194] Registered new tserver with Master: c55cf3a0245f4282a606a576da7f8c6d (127.0.97.2:40787)
I20260924 19:03:14.101936 14269 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44707
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 19:03:14.137020 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45755
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:03:14.456840 14597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:03:14.457183 14597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:03:14.457355 14597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:03:14.464651 14597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:03:14.464828 14597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:03:14.465011 14597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:03:14.474619 14597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:03:14.475919 14597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:03:14.477218 14597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:03:14.490456 14602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:14.490507 14605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:14.490418 14603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:03:14.491221 14597 server_base.cc:1034] running on GCE node
I20260924 19:03:14.491863 14597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:03:14.492805 14597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:03:14.494028 14597 hybrid_clock.cc:648] HybridClock initialized: now 1790276594493968 us; error 53 us; skew 500 ppm
I20260924 19:03:14.494354 14597 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:03:14 dist-test-slave-9fn1 krb5kdc[14224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:03:14.497596 14597 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 19:03:14.498966 14597 webserver.cc:460] Webserver started at http://127.0.97.3:32927/ using document root <none> and password file <none>
I20260924 19:03:14.499639 14597 fs_manager.cc:359] Metadata directory not provided
I20260924 19:03:14.499782 14597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:03:14.500092 14597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:03:14.503183 14597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/data/instance:
uuid: "dabdbce8d75e497192b5effcced1c154"
format_stamp: "Formatted at 2026-09-24 19:03:14 on dist-test-slave-9fn1"
server_key: "0d80a45c3f85154140e6c12b762766f3"
server_key_iv: "c37e00dbb0cc7cd5261af5ab0988f023"
server_key_version: "encryptionkey@0"
I20260924 19:03:14.503945 14597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/wal/instance:
uuid: "dabdbce8d75e497192b5effcced1c154"
format_stamp: "Formatted at 2026-09-24 19:03:14 on dist-test-slave-9fn1"
server_key: "0d80a45c3f85154140e6c12b762766f3"
server_key_iv: "c37e00dbb0cc7cd5261af5ab0988f023"
server_key_version: "encryptionkey@0"
I20260924 19:03:14.509004 14597 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 19:03:14.512432 14612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:14.513893 14597 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:03:14.514065 14597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "dabdbce8d75e497192b5effcced1c154"
format_stamp: "Formatted at 2026-09-24 19:03:14 on dist-test-slave-9fn1"
server_key: "0d80a45c3f85154140e6c12b762766f3"
server_key_iv: "c37e00dbb0cc7cd5261af5ab0988f023"
server_key_version: "encryptionkey@0"
I20260924 19:03:14.514259 14597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:03:14.532656 14597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:03:14.536626 14597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:03:14.536921 14597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:03:14.537933 14597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:03:14.539723 14597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:03:14.539824 14597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:14.539969 14597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:03:14.540055 14597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:14.572541 14597 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39891
I20260924 19:03:14.572593 14724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39891 every 8 connection(s)
I20260924 19:03:14.573844 14597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:03:14.578454 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 14597
I20260924 19:03:14.578642 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/wal/instance
I20260924 19:03:14.579087 388 external_mini_cluster.cc:1428] Setting key 27aa8e7615af3f6b6acceb015c0d4cd9
Sep 24 19:03:14 dist-test-slave-9fn1 krb5kdc[14224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:03:14.592425 14458 heartbeater.cc:503] Master 127.0.97.62:45755 was elected leader, sending a full tablet report...
I20260924 19:03:14.598285 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:14.576398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52525 (local address 127.0.97.62:45755)
0924 19:03:14.576576 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:14.576580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:14.577661 (+ 1081us) server_negotiation.cc:407] Connection header received
0924 19:03:14.579712 (+ 2051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:14.579718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:14.579816 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:14.579915 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:14.582272 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.583616 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.586049 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.586562 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.586704 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:14.590894 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:14.590907 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:14.590911 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:14.590950 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:14.593743 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:14.594717 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:14.594722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:14.594726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:14.594874 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:14.595436 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:14.595441 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:14.595445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:14.595879 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:14.596058 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:14.597706 (+ 1648us) server_negotiation.cc:299] Negotiation successful
0924 19:03:14.597928 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 19:03:14.598788 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:14.576775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.3:52525)
0924 19:03:14.577455 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:14.577491 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:14.579325 (+ 1834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:14.580342 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:14.580354 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:14.581156 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:03:14.582069 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:14.582095 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.583812 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.583820 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:14.585899 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:14.585911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.586831 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.586840 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:14.587175 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:14.588015 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:03:14.588058 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:03:14.590676 (+ 2618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:03:14.593903 (+ 3227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:14.593914 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:14.593933 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:14.594561 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:14.595039 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:14.595043 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:14.595046 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:14.595298 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:14.596018 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:03:14.596027 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:03:14.596476 (+ 449us) client_negotiation.cc:769] Sending connection context
0924 19:03:14.596815 (+ 339us) client_negotiation.cc:240] Negotiation successful
0924 19:03:14.597442 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":269,"threads_started":1}
I20260924 19:03:14.600054 14725 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45755
I20260924 19:03:14.600364 14725 heartbeater.cc:464] Registering TS with master...
I20260924 19:03:14.601073 14725 heartbeater.cc:511] Master 127.0.97.62:45755 requested a full tablet report, sending...
I20260924 19:03:14.602568 14269 ts_manager.cc:194] Registered new tserver with Master: dabdbce8d75e497192b5effcced1c154 (127.0.97.3:39891)
I20260924 19:03:14.603410 14269 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:52525
I20260924 19:03:14.606947 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:03:14.623260 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:14.611322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44468 (local address 127.0.97.62:45755)
0924 19:03:14.611466 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:14.611471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:14.611976 (+ 505us) server_negotiation.cc:407] Connection header received
0924 19:03:14.612110 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:14.612113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:14.612167 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:14.612242 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:14.613715 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.614384 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.615582 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.615933 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.616023 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:14.618376 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:14.618380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:14.618383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:14.618412 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:14.621027 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:14.621683 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:14.621687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:14.621689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:14.621799 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:14.622124 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:14.622128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:14.622129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:14.622418 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:14.622601 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:14.622870 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 19:03:14.623021 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 19:03:14.627074 14269 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:03:14.633904 14269 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:03:14.663066 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:14.646901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37210 (local address 127.0.97.1:39497)
0924 19:03:14.654001 (+ 7100us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:14.654010 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:14.655175 (+ 1165us) server_negotiation.cc:407] Connection header received
0924 19:03:14.655459 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:14.655466 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:14.655728 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:14.655895 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:03:14.656948 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.658449 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.660939 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.661922 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.662070 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:14.662299 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:14.662571 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 19:03:14.662741 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6909,"thread_start_us":6973,"threads_started":1}
I20260924 19:03:14.663211 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:14.645276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39891 (local address 127.0.0.1:58636)
0924 19:03:14.646023 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:14.646059 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:14.646225 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:14.655356 (+ 9131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:14.655363 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:14.655414 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:03:14.655911 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:14.655932 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.657929 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.657936 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:14.660510 (+ 2574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:14.660524 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.662211 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.662217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:14.662397 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:14.662443 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 19:03:14.662512 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 19:03:14.662729 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":97,"spinlock_wait_cycles":18048,"thread_start_us":390,"threads_started":1}
I20260924 19:03:14.663251 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:14.645832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40787 (local address 127.0.0.1:41372)
0924 19:03:14.653668 (+ 7836us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:14.653691 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:14.653816 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:14.655365 (+ 1549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:14.655369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:14.655407 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:03:14.656494 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:14.656510 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.658149 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.658155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:14.660298 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:14.660310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.662415 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.662434 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:14.662599 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:14.662632 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 19:03:14.662707 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 19:03:14.662814 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7686,"mutex_wait_us":30,"spinlock_wait_cycles":896,"thread_start_us":8975,"threads_started":1}
I20260924 19:03:14.663590 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:14.646150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58636 (local address 127.0.97.3:39891)
0924 19:03:14.654528 (+ 8378us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:14.654536 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:14.654562 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:03:14.654643 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:14.654651 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:14.655125 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:14.655674 (+ 549us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:03:14.656119 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.657772 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.660661 (+ 2889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.661978 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.662837 (+ 859us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:14.663045 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:14.663123 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 19:03:14.663286 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8181,"thread_start_us":8268,"threads_started":1}
I20260924 19:03:14.664012 14738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:14.646429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39497 (local address 127.0.0.1:37210)
0924 19:03:14.655069 (+ 8640us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:14.655086 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:14.655271 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:14.656216 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:14.656220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:14.656257 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:03:14.656760 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:14.656774 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.658598 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.658603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:14.660805 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:14.660814 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.662107 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.662113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:14.662308 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:14.662370 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 19:03:14.662532 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 19:03:14.662730 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8519,"mutex_wait_us":44,"thread_start_us":6949,"threads_started":1}
I20260924 19:03:14.664162 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:14.645884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41372 (local address 127.0.97.2:40787)
0924 19:03:14.654478 (+ 8594us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:14.654503 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:14.654534 (+ 31us) server_negotiation.cc:407] Connection header received
0924 19:03:14.654643 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:14.654651 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:14.654922 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:14.655116 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:03:14.656727 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.657996 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.660447 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.661940 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.662084 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:14.662281 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:14.663458 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0924 19:03:14.663636 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8429,"thread_start_us":8326,"threads_started":1}
I20260924 19:03:14.667287 14527 tablet_service.cc:1467] Processing CreateTablet for tablet 77009948c9ab4405af7bcc7692a7e5dd (DEFAULT_TABLE table=test-table [id=3ddcbc7a6f074307a97aac1c08917ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:03:14.667542 14660 tablet_service.cc:1467] Processing CreateTablet for tablet 77009948c9ab4405af7bcc7692a7e5dd (DEFAULT_TABLE table=test-table [id=3ddcbc7a6f074307a97aac1c08917ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:03:14.667759 14393 tablet_service.cc:1467] Processing CreateTablet for tablet 77009948c9ab4405af7bcc7692a7e5dd (DEFAULT_TABLE table=test-table [id=3ddcbc7a6f074307a97aac1c08917ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:03:14.668504 14527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77009948c9ab4405af7bcc7692a7e5dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:03:14.668888 14660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77009948c9ab4405af7bcc7692a7e5dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:03:14.668905 14393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77009948c9ab4405af7bcc7692a7e5dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:03:14.680188 14743 tablet_bootstrap.cc:492] T 77009948c9ab4405af7bcc7692a7e5dd P c55cf3a0245f4282a606a576da7f8c6d: Bootstrap starting.
I20260924 19:03:14.683230 14744 tablet_bootstrap.cc:492] T 77009948c9ab4405af7bcc7692a7e5dd P dabdbce8d75e497192b5effcced1c154: Bootstrap starting.
I20260924 19:03:14.683486 14743 tablet_bootstrap.cc:654] T 77009948c9ab4405af7bcc7692a7e5dd P c55cf3a0245f4282a606a576da7f8c6d: Neither blocks nor log segments found. Creating new log.
I20260924 19:03:14.684839 14743 log.cc:824] T 77009948c9ab4405af7bcc7692a7e5dd P c55cf3a0245f4282a606a576da7f8c6d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:03:14.684880 14745 tablet_bootstrap.cc:492] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8: Bootstrap starting.
I20260924 19:03:14.687997 14745 tablet_bootstrap.cc:654] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8: Neither blocks nor log segments found. Creating new log.
I20260924 19:03:14.687985 14744 tablet_bootstrap.cc:654] T 77009948c9ab4405af7bcc7692a7e5dd P dabdbce8d75e497192b5effcced1c154: Neither blocks nor log segments found. Creating new log.
I20260924 19:03:14.689069 14745 log.cc:824] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8: Log is configured to *not* fsync() on all Append() calls
I20260924 19:03:14.689503 14744 log.cc:824] T 77009948c9ab4405af7bcc7692a7e5dd P dabdbce8d75e497192b5effcced1c154: Log is configured to *not* fsync() on all Append() calls
I20260924 19:03:14.691828 14745 tablet_bootstrap.cc:492] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8: No bootstrap required, opened a new log
I20260924 19:03:14.692072 14745 ts_tablet_manager.cc:1397] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 19:03:14.692147 14743 tablet_bootstrap.cc:492] T 77009948c9ab4405af7bcc7692a7e5dd P c55cf3a0245f4282a606a576da7f8c6d: No bootstrap required, opened a new log
I20260924 19:03:14.692390 14744 tablet_bootstrap.cc:492] T 77009948c9ab4405af7bcc7692a7e5dd P dabdbce8d75e497192b5effcced1c154: No bootstrap required, opened a new log
I20260924 19:03:14.692391 14743 ts_tablet_manager.cc:1397] T 77009948c9ab4405af7bcc7692a7e5dd P c55cf3a0245f4282a606a576da7f8c6d: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260924 19:03:14.692603 14744 ts_tablet_manager.cc:1397] T 77009948c9ab4405af7bcc7692a7e5dd P dabdbce8d75e497192b5effcced1c154: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260924 19:03:14.699534 14745 raft_consensus.cc:348] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbce8d75e497192b5effcced1c154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39891 } } peers { permanent_uuid: "fc0a539cb7784d15a1150b61ad525ce8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39497 } } peers { permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 } }
I20260924 19:03:14.699885 14745 raft_consensus.cc:374] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:03:14.700026 14745 raft_consensus.cc:729] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc0a539cb7784d15a1150b61ad525ce8, State: Initialized, Role: FOLLOWER
I20260924 19:03:14.700627 14745 consensus_queue.cc:260] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbce8d75e497192b5effcced1c154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39891 } } peers { permanent_uuid: "fc0a539cb7784d15a1150b61ad525ce8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39497 } } peers { permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 } }
I20260924 19:03:14.702371 14745 ts_tablet_manager.cc:1428] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260924 19:03:14.703109 14744 raft_consensus.cc:348] T 77009948c9ab4405af7bcc7692a7e5dd P dabdbce8d75e497192b5effcced1c154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbce8d75e497192b5effcced1c154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39891 } } peers { permanent_uuid: "fc0a539cb7784d15a1150b61ad525ce8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39497 } } peers { permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 } }
I20260924 19:03:14.703096 14743 raft_consensus.cc:348] T 77009948c9ab4405af7bcc7692a7e5dd P c55cf3a0245f4282a606a576da7f8c6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbce8d75e497192b5effcced1c154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39891 } } peers { permanent_uuid: "fc0a539cb7784d15a1150b61ad525ce8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39497 } } peers { permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 } }
I20260924 19:03:14.703504 14743 raft_consensus.cc:374] T 77009948c9ab4405af7bcc7692a7e5dd P c55cf3a0245f4282a606a576da7f8c6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:03:14.703521 14744 raft_consensus.cc:374] T 77009948c9ab4405af7bcc7692a7e5dd P dabdbce8d75e497192b5effcced1c154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:03:14.703619 14744 raft_consensus.cc:729] T 77009948c9ab4405af7bcc7692a7e5dd P dabdbce8d75e497192b5effcced1c154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dabdbce8d75e497192b5effcced1c154, State: Initialized, Role: FOLLOWER
I20260924 19:03:14.703619 14743 raft_consensus.cc:729] T 77009948c9ab4405af7bcc7692a7e5dd P c55cf3a0245f4282a606a576da7f8c6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c55cf3a0245f4282a606a576da7f8c6d, State: Initialized, Role: FOLLOWER
I20260924 19:03:14.704202 14744 consensus_queue.cc:260] T 77009948c9ab4405af7bcc7692a7e5dd P dabdbce8d75e497192b5effcced1c154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbce8d75e497192b5effcced1c154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39891 } } peers { permanent_uuid: "fc0a539cb7784d15a1150b61ad525ce8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39497 } } peers { permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 } }
I20260924 19:03:14.704205 14743 consensus_queue.cc:260] T 77009948c9ab4405af7bcc7692a7e5dd P c55cf3a0245f4282a606a576da7f8c6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbce8d75e497192b5effcced1c154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39891 } } peers { permanent_uuid: "fc0a539cb7784d15a1150b61ad525ce8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39497 } } peers { permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 } }
I20260924 19:03:14.705348 14725 heartbeater.cc:503] Master 127.0.97.62:45755 was elected leader, sending a full tablet report...
I20260924 19:03:14.705775 14592 heartbeater.cc:503] Master 127.0.97.62:45755 was elected leader, sending a full tablet report...
I20260924 19:03:14.706036 14744 ts_tablet_manager.cc:1428] T 77009948c9ab4405af7bcc7692a7e5dd P dabdbce8d75e497192b5effcced1c154: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260924 19:03:14.707554 14743 ts_tablet_manager.cc:1428] T 77009948c9ab4405af7bcc7692a7e5dd P c55cf3a0245f4282a606a576da7f8c6d: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260924 19:03:14.709481 14749 raft_consensus.cc:482] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:03:14.709764 14749 raft_consensus.cc:504] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbce8d75e497192b5effcced1c154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39891 } } peers { permanent_uuid: "fc0a539cb7784d15a1150b61ad525ce8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39497 } } peers { permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 } }
I20260924 19:03:14.711628 14749 leader_election.cc:288] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dabdbce8d75e497192b5effcced1c154 (127.0.97.3:39891), c55cf3a0245f4282a606a576da7f8c6d (127.0.97.2:40787)
I20260924 19:03:14.719565 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:14.712561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57531 (local address 127.0.97.3:39891)
0924 19:03:14.712751 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:14.712755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:14.712860 (+ 105us) server_negotiation.cc:407] Connection header received
0924 19:03:14.713001 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:14.713009 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:14.713112 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:14.713223 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:03:14.713845 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.715324 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.716979 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.718456 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.718575 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:14.718674 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:14.719200 (+ 526us) server_negotiation.cc:299] Negotiation successful
0924 19:03:14.719319 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 19:03:14.719594 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:14.711993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39891 (local address 127.0.97.1:57531)
0924 19:03:14.712749 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:14.712770 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:14.712896 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:14.713276 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:14.713281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:14.713315 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:03:14.713681 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:14.713708 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.715460 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.715464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:14.716831 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:14.716842 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.718727 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.718736 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:14.718959 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:14.719006 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 19:03:14.719177 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 19:03:14.719370 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":448,"threads_started":1}
I20260924 19:03:14.720857 14680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77009948c9ab4405af7bcc7692a7e5dd" candidate_uuid: "fc0a539cb7784d15a1150b61ad525ce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabdbce8d75e497192b5effcced1c154" is_pre_election: true
I20260924 19:03:14.721388 14680 raft_consensus.cc:2454] T 77009948c9ab4405af7bcc7692a7e5dd P dabdbce8d75e497192b5effcced1c154 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc0a539cb7784d15a1150b61ad525ce8 in term 0.
I20260924 19:03:14.722046 14347 leader_election.cc:302] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dabdbce8d75e497192b5effcced1c154, fc0a539cb7784d15a1150b61ad525ce8; no voters:
I20260924 19:03:14.722450 14749 raft_consensus.cc:2781] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:03:14.722600 14749 raft_consensus.cc:482] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:03:14.722716 14749 raft_consensus.cc:3037] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:03:14.724185 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:14.712568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33135 (local address 127.0.97.2:40787)
0924 19:03:14.712747 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:14.712753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:14.718195 (+ 5442us) server_negotiation.cc:407] Connection header received
0924 19:03:14.718368 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:14.718375 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:14.718463 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:14.718552 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:03:14.719278 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.720226 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.722376 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.723342 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.723444 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:14.723508 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:14.723916 (+ 408us) server_negotiation.cc:299] Negotiation successful
0924 19:03:14.723999 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 19:03:14.724447 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:14.712026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40787 (local address 127.0.97.1:33135)
0924 19:03:14.718096 (+ 6070us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:14.718111 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:14.718248 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:14.718607 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:14.718611 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:14.718641 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:03:14.719068 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:14.719079 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.720373 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.720379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:14.722179 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:14.722192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.723554 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.723561 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:14.723751 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:14.723794 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 19:03:14.723890 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 19:03:14.724017 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5925,"thread_start_us":5767,"threads_started":1}
I20260924 19:03:14.725548 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77009948c9ab4405af7bcc7692a7e5dd" candidate_uuid: "fc0a539cb7784d15a1150b61ad525ce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c55cf3a0245f4282a606a576da7f8c6d" is_pre_election: true
I20260924 19:03:14.725754 14749 raft_consensus.cc:504] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbce8d75e497192b5effcced1c154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39891 } } peers { permanent_uuid: "fc0a539cb7784d15a1150b61ad525ce8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39497 } } peers { permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 } }
I20260924 19:03:14.726035 14547 raft_consensus.cc:2454] T 77009948c9ab4405af7bcc7692a7e5dd P c55cf3a0245f4282a606a576da7f8c6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc0a539cb7784d15a1150b61ad525ce8 in term 0.
I20260924 19:03:14.726473 14749 leader_election.cc:288] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8 [CANDIDATE]: Term 1 election: Requested vote from peers dabdbce8d75e497192b5effcced1c154 (127.0.97.3:39891), c55cf3a0245f4282a606a576da7f8c6d (127.0.97.2:40787)
I20260924 19:03:14.726814 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77009948c9ab4405af7bcc7692a7e5dd" candidate_uuid: "fc0a539cb7784d15a1150b61ad525ce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c55cf3a0245f4282a606a576da7f8c6d"
I20260924 19:03:14.726845 14680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77009948c9ab4405af7bcc7692a7e5dd" candidate_uuid: "fc0a539cb7784d15a1150b61ad525ce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabdbce8d75e497192b5effcced1c154"
I20260924 19:03:14.726989 14547 raft_consensus.cc:3037] T 77009948c9ab4405af7bcc7692a7e5dd P c55cf3a0245f4282a606a576da7f8c6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:03:14.727026 14680 raft_consensus.cc:3037] T 77009948c9ab4405af7bcc7692a7e5dd P dabdbce8d75e497192b5effcced1c154 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:03:14.729075 14547 raft_consensus.cc:2454] T 77009948c9ab4405af7bcc7692a7e5dd P c55cf3a0245f4282a606a576da7f8c6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc0a539cb7784d15a1150b61ad525ce8 in term 1.
I20260924 19:03:14.729112 14680 raft_consensus.cc:2454] T 77009948c9ab4405af7bcc7692a7e5dd P dabdbce8d75e497192b5effcced1c154 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc0a539cb7784d15a1150b61ad525ce8 in term 1.
I20260924 19:03:14.729534 14347 leader_election.cc:302] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dabdbce8d75e497192b5effcced1c154, fc0a539cb7784d15a1150b61ad525ce8; no voters:
I20260924 19:03:14.729810 14749 raft_consensus.cc:2781] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:03:14.730212 14749 raft_consensus.cc:686] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8 [term 1 LEADER]: Becoming Leader. State: Replica: fc0a539cb7784d15a1150b61ad525ce8, State: Running, Role: LEADER
I20260924 19:03:14.730643 14749 consensus_queue.cc:237] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbce8d75e497192b5effcced1c154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39891 } } peers { permanent_uuid: "fc0a539cb7784d15a1150b61ad525ce8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39497 } } peers { permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 } }
I20260924 19:03:14.734941 14268 catalog_manager.cc:5495] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc0a539cb7784d15a1150b61ad525ce8 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "fc0a539cb7784d15a1150b61ad525ce8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dabdbce8d75e497192b5effcced1c154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc0a539cb7784d15a1150b61ad525ce8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 } health_report { overall_health: UNKNOWN } } }
I20260924 19:03:14.757421 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:14.752006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37212 (local address 127.0.97.1:39497)
0924 19:03:14.752207 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:14.752214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:14.752266 (+ 52us) server_negotiation.cc:407] Connection header received
0924 19:03:14.752524 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:14.752527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:14.752617 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:14.752770 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:03:14.753164 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.753955 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.755335 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.755731 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.755860 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:14.756363 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:03:14.756824 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:03:14.756994 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:14.757093 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 19:03:14.757172 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 19:03:14.764998 14680 raft_consensus.cc:1261] T 77009948c9ab4405af7bcc7692a7e5dd P dabdbce8d75e497192b5effcced1c154 [term 1 FOLLOWER]: Refusing update from remote peer fc0a539cb7784d15a1150b61ad525ce8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:03:14.764998 14547 raft_consensus.cc:1261] T 77009948c9ab4405af7bcc7692a7e5dd P c55cf3a0245f4282a606a576da7f8c6d [term 1 FOLLOWER]: Refusing update from remote peer fc0a539cb7784d15a1150b61ad525ce8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:03:14.765930 14754 consensus_queue.cc:1035] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:03:14.766276 14749 consensus_queue.cc:1035] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabdbce8d75e497192b5effcced1c154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:03:14.781699 14758 mvcc.cc:205] Tried to move back new op lower bound from 7332972932148445184 to 7332972932020101120. Current Snapshot: MvccSnapshot[applied={T|T < 7332972932148445184}]
I20260924 19:03:14.782250 14759 mvcc.cc:205] Tried to move back new op lower bound from 7332972932148445184 to 7332972932020101120. Current Snapshot: MvccSnapshot[applied={T|T < 7332972932148445184}]
I20260924 19:03:14.790458 14268 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:03:14.790726 14268 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:03:14.794368 14268 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b: Sending DeleteTablet for 3 replicas of tablet 77009948c9ab4405af7bcc7692a7e5dd
I20260924 19:03:14.795596 14393 tablet_service.cc:1514] Processing DeleteTablet for tablet 77009948c9ab4405af7bcc7692a7e5dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:03:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37210
I20260924 19:03:14.795576 14527 tablet_service.cc:1514] Processing DeleteTablet for tablet 77009948c9ab4405af7bcc7692a7e5dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:03:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41372
I20260924 19:03:14.796352 14660 tablet_service.cc:1514] Processing DeleteTablet for tablet 77009948c9ab4405af7bcc7692a7e5dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:03:14 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58636
I20260924 19:03:14.796494 14766 tablet_replica.cc:330] T 77009948c9ab4405af7bcc7692a7e5dd P c55cf3a0245f4282a606a576da7f8c6d: stopping tablet replica
I20260924 19:03:14.796970 14766 raft_consensus.cc:2229] T 77009948c9ab4405af7bcc7692a7e5dd P c55cf3a0245f4282a606a576da7f8c6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:03:14.797565 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 14330
I20260924 19:03:14.797559 14766 raft_consensus.cc:2258] T 77009948c9ab4405af7bcc7692a7e5dd P c55cf3a0245f4282a606a576da7f8c6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:03:14.799742 14766 ts_tablet_manager.cc:1905] T 77009948c9ab4405af7bcc7692a7e5dd P c55cf3a0245f4282a606a576da7f8c6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:03:14.800375 14767 tablet_replica.cc:330] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8: stopping tablet replica
I20260924 19:03:14.801831 14767 raft_consensus.cc:2229] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:03:14.806916 14766 ts_tablet_manager.cc:1918] T 77009948c9ab4405af7bcc7692a7e5dd P c55cf3a0245f4282a606a576da7f8c6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:03:14.807147 14766 log.cc:1196] T 77009948c9ab4405af7bcc7692a7e5dd P c55cf3a0245f4282a606a576da7f8c6d: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/wal/wals/77009948c9ab4405af7bcc7692a7e5dd
I20260924 19:03:14.807760 14766 ts_tablet_manager.cc:1939] T 77009948c9ab4405af7bcc7692a7e5dd P c55cf3a0245f4282a606a576da7f8c6d: Deleting consensus metadata
I20260924 19:03:14.808097 14768 tablet_replica.cc:330] T 77009948c9ab4405af7bcc7692a7e5dd P dabdbce8d75e497192b5effcced1c154: stopping tablet replica
I20260924 19:03:14.808705 14768 raft_consensus.cc:2229] T 77009948c9ab4405af7bcc7692a7e5dd P dabdbce8d75e497192b5effcced1c154 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:03:14.808867 14767 raft_consensus.cc:2258] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:03:14.809299 14768 raft_consensus.cc:2258] T 77009948c9ab4405af7bcc7692a7e5dd P dabdbce8d75e497192b5effcced1c154 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:03:14.809859 14255 catalog_manager.cc:4841] TS c55cf3a0245f4282a606a576da7f8c6d (127.0.97.2:40787): tablet 77009948c9ab4405af7bcc7692a7e5dd (table test-table [id=3ddcbc7a6f074307a97aac1c08917ec9]) successfully deleted
I20260924 19:03:14.810925 14767 ts_tablet_manager.cc:1905] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:03:14.811928 14768 ts_tablet_manager.cc:1905] T 77009948c9ab4405af7bcc7692a7e5dd P dabdbce8d75e497192b5effcced1c154: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 19:03:14 dist-test-slave-9fn1 krb5kdc[14224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 19:03:14.817427 14768 ts_tablet_manager.cc:1918] T 77009948c9ab4405af7bcc7692a7e5dd P dabdbce8d75e497192b5effcced1c154: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:03:14.817577 14767 ts_tablet_manager.cc:1918] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:03:14.817621 14768 log.cc:1196] T 77009948c9ab4405af7bcc7692a7e5dd P dabdbce8d75e497192b5effcced1c154: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/wal/wals/77009948c9ab4405af7bcc7692a7e5dd
I20260924 19:03:14.817744 14767 log.cc:1196] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/wal/wals/77009948c9ab4405af7bcc7692a7e5dd
I20260924 19:03:14.818156 14768 ts_tablet_manager.cc:1939] T 77009948c9ab4405af7bcc7692a7e5dd P dabdbce8d75e497192b5effcced1c154: Deleting consensus metadata
I20260924 19:03:14.818272 14767 ts_tablet_manager.cc:1939] T 77009948c9ab4405af7bcc7692a7e5dd P fc0a539cb7784d15a1150b61ad525ce8: Deleting consensus metadata
I20260924 19:03:14.819860 14254 catalog_manager.cc:4841] TS fc0a539cb7784d15a1150b61ad525ce8 (127.0.97.1:39497): tablet 77009948c9ab4405af7bcc7692a7e5dd (table test-table [id=3ddcbc7a6f074307a97aac1c08917ec9]) successfully deleted
I20260924 19:03:14.820297 14254 catalog_manager.cc:4841] TS dabdbce8d75e497192b5effcced1c154 (127.0.97.3:39891): tablet 77009948c9ab4405af7bcc7692a7e5dd (table test-table [id=3ddcbc7a6f074307a97aac1c08917ec9]) successfully deleted
I20260924 19:03:14.821872 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:14.799018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37222 (local address 127.0.97.1:39497)
0924 19:03:14.799206 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:14.799214 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:14.799235 (+ 21us) server_negotiation.cc:407] Connection header received
0924 19:03:14.799479 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:14.799488 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:14.799595 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:14.799686 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:14.802376 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.804013 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.806805 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.807772 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.807880 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:14.814086 (+ 6206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:14.814098 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:14.814115 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:14.814162 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:14.818361 (+ 4199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:14.819302 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:14.819310 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:14.819313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:14.819500 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:14.819953 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:14.819961 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:14.819966 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:14.820446 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:14.820679 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:14.821240 (+ 561us) server_negotiation.cc:299] Negotiation successful
0924 19:03:14.821478 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 19:03:14.822602 14453 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:03:14.917546 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 14330
I20260924 19:03:14.935966 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 14464
Sep 24 19:03:14 dist-test-slave-9fn1 krb5kdc[14224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 19:03:14.951975 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:14.936774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41384 (local address 127.0.97.2:40787)
0924 19:03:14.936955 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:14.936960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:14.936996 (+ 36us) server_negotiation.cc:407] Connection header received
0924 19:03:14.937050 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:14.937061 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:14.937130 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:14.937203 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:14.938462 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.939203 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.940441 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:14.941071 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:14.941188 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:14.945529 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:14.945548 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:14.945576 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:14.945616 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:14.949079 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:14.949929 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:14.949939 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:14.949943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:14.950088 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:14.950455 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:14.950460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:14.950463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:14.950898 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:14.951175 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:14.951403 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 19:03:14.951634 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 19:03:14.952696 14587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:03:15.041929 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 14464
I20260924 19:03:15.058708 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 14597
Sep 24 19:03:15 dist-test-slave-9fn1 krb5kdc[14224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 19:03:15.073294 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:15.059525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58650 (local address 127.0.97.3:39891)
0924 19:03:15.059694 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:15.059698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:15.059717 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:03:15.059838 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:15.059842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:15.059904 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:15.059997 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:15.061438 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:15.062220 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:15.063524 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:15.063890 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:15.063978 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:15.068209 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:15.068223 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:15.068239 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:15.068271 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:15.071096 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:15.071655 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:15.071662 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:15.071664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:15.071781 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:15.072114 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:15.072117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:15.072119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:15.072441 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:15.072647 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:15.072880 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 19:03:15.073031 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 19:03:15.073974 14720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:03:15.163357 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 14597
I20260924 19:03:15.180243 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 14237
I20260924 19:03:15.181331 14299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:03:15.289980 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 14237
I20260924 19:03:15.311281 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45755
--webserver_interface=127.0.97.62
--webserver_port=35637
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:03:15.628401 14774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:03:15.628672 14774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:03:15.628786 14774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:03:15.635826 14774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:03:15.635938 14774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:03:15.635990 14774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:03:15.636070 14774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:03:15.636138 14774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:03:15.645522 14774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=35637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:03:15.646754 14774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:03:15.648025 14774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:03:15.669514 14780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:15.669557 14781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:15.669482 14783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:03:15.670569 14774 server_base.cc:1034] running on GCE node
I20260924 19:03:15.671451 14774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:03:15.673035 14774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:03:15.674237 14774 hybrid_clock.cc:648] HybridClock initialized: now 1790276595674190 us; error 33 us; skew 500 ppm
I20260924 19:03:15.674638 14774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:03:15 dist-test-slave-9fn1 krb5kdc[14224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:03:15.678062 14774 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:03:15.679674 14774 webserver.cc:460] Webserver started at http://127.0.97.62:35637/ using document root <none> and password file <none>
I20260924 19:03:15.680408 14774 fs_manager.cc:359] Metadata directory not provided
I20260924 19:03:15.680564 14774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:03:15.686307 14774 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 19:03:15.689129 14790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:15.690367 14774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 19:03:15.690551 14774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "8e301c22a72249bc9d55d0eda728c70b"
format_stamp: "Formatted at 2026-09-24 19:03:12 on dist-test-slave-9fn1"
server_key: "23db884e266ad7796146c591501f6a3d"
server_key_iv: "8a931e9f2fff36850c1a5ad7ee20c339"
server_key_version: "encryptionkey@0"
I20260924 19:03:15.691622 14774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:03:15.716554 14774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:03:15.721264 14774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:03:15.721655 14774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:03:15.740617 14774 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45755
I20260924 19:03:15.740643 14841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45755 every 8 connection(s)
I20260924 19:03:15.742285 14774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:03:15.743113 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 14774
I20260924 19:03:15.744052 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39497
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45755
--webserver_port=33025
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:03:15.749709 14842 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:03:15.755086 14842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b: Bootstrap starting.
I20260924 19:03:15.785284 14842 log.cc:824] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b: Log is configured to *not* fsync() on all Append() calls
I20260924 19:03:15.800663 14842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:03:15.801399 14842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b: Bootstrap complete.
I20260924 19:03:15.816779 14842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e301c22a72249bc9d55d0eda728c70b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45755 } }
I20260924 19:03:15.818015 14842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e301c22a72249bc9d55d0eda728c70b, State: Initialized, Role: FOLLOWER
I20260924 19:03:15.818758 14842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e301c22a72249bc9d55d0eda728c70b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45755 } }
I20260924 19:03:15.819054 14842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:03:15.819178 14842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:03:15.819355 14842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:03:15.824892 14842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e301c22a72249bc9d55d0eda728c70b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45755 } }
I20260924 19:03:15.825541 14842 leader_election.cc:302] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e301c22a72249bc9d55d0eda728c70b; no voters:
I20260924 19:03:15.826179 14842 leader_election.cc:288] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 19:03:15.826323 14846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:03:15.827497 14846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [term 2 LEADER]: Becoming Leader. State: Replica: 8e301c22a72249bc9d55d0eda728c70b, State: Running, Role: LEADER
I20260924 19:03:15.827998 14846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e301c22a72249bc9d55d0eda728c70b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45755 } }
I20260924 19:03:15.828306 14842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:03:15.830652 14848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e301c22a72249bc9d55d0eda728c70b. Latest consensus state: current_term: 2 leader_uuid: "8e301c22a72249bc9d55d0eda728c70b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e301c22a72249bc9d55d0eda728c70b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45755 } } }
I20260924 19:03:15.830781 14847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8e301c22a72249bc9d55d0eda728c70b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e301c22a72249bc9d55d0eda728c70b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45755 } } }
I20260924 19:03:15.831053 14847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [sys.catalog]: This master's current role is: LEADER
I20260924 19:03:15.831051 14848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [sys.catalog]: This master's current role is: LEADER
I20260924 19:03:15.844337 14853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:03:15.852295 14853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:03:15.853758 14853 catalog_manager.cc:1259] Loaded cluster ID: ab8033854edd473da4e1ed318474d999
I20260924 19:03:15.853853 14853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:03:15.858206 14853 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 19:03:15.871857 14863 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 19:03:15.876242 14853 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 19:03:15.885182 14853 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:03:15.886868 14853 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b: Loaded TSK: 0
I20260924 19:03:15.888366 14853 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 19:03:16.111739 14844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:03:16.112049 14844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:03:16.112215 14844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:03:16.119506 14844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:03:16.119656 14844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:03:16.119825 14844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:03:16.129738 14844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=33025
--tserver_master_addrs=127.0.97.62:45755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:03:16.131043 14844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:03:16.132326 14844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:03:16.147110 14870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:16.147708 14872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:16.147796 14869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:03:16.148200 14844 server_base.cc:1034] running on GCE node
I20260924 19:03:16.148864 14844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:03:16.150012 14844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:03:16.151250 14844 hybrid_clock.cc:648] HybridClock initialized: now 1790276596151187 us; error 54 us; skew 500 ppm
I20260924 19:03:16.151578 14844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:03:16 dist-test-slave-9fn1 krb5kdc[14224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:03:16.154973 14844 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 19:03:16.156628 14844 webserver.cc:460] Webserver started at http://127.0.97.1:33025/ using document root <none> and password file <none>
I20260924 19:03:16.157326 14844 fs_manager.cc:359] Metadata directory not provided
I20260924 19:03:16.157487 14844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:03:16.163051 14844 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260924 19:03:16.165872 14879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:16.167295 14844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 19:03:16.167474 14844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "fc0a539cb7784d15a1150b61ad525ce8"
format_stamp: "Formatted at 2026-09-24 19:03:13 on dist-test-slave-9fn1"
server_key: "81deae84fae9525c71a42bb55270b7ce"
server_key_iv: "0e0de2429461d95bf0dab6a086a14d59"
server_key_version: "encryptionkey@0"
I20260924 19:03:16.168453 14844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:03:16.188153 14844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:03:16.192533 14844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:03:16.192816 14844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:03:16.193851 14844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:03:16.195349 14844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:03:16.195479 14844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:16.195631 14844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:03:16.195720 14844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:16.229138 14844 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39497
I20260924 19:03:16.229241 14991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39497 every 8 connection(s)
I20260924 19:03:16.230639 14844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:03:16.240343 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 14844
I20260924 19:03:16.240964 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40787
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45755
--webserver_port=33037
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:03:16 dist-test-slave-9fn1 krb5kdc[14224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:03:16.254038 14995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:16.233467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.1:50853)
0924 19:03:16.234337 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:16.234387 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:16.236060 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:16.237277 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:16.237284 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:16.237866 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:03:16.238398 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:16.238416 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:16.239985 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:16.239989 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:16.241346 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:16.241358 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:16.242839 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:16.242843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:16.243021 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:16.243696 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:03:16.243722 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:03:16.246054 (+ 2332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:03:16.250281 (+ 4227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:16.250287 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:16.250297 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:16.250683 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:16.251167 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:16.251170 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:16.251173 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:16.251332 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:16.252240 (+ 908us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:03:16.252245 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:03:16.252503 (+ 258us) client_negotiation.cc:769] Sending connection context
0924 19:03:16.252704 (+ 201us) client_negotiation.cc:240] Negotiation successful
0924 19:03:16.253040 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":345,"threads_started":1}
I20260924 19:03:16.254608 14994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:16.233347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50853 (local address 127.0.97.62:45755)
0924 19:03:16.233967 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:16.233979 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:16.234505 (+ 526us) server_negotiation.cc:407] Connection header received
0924 19:03:16.236517 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:16.236529 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:16.236830 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:16.237137 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:16.238591 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:16.239801 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:16.241482 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:16.242635 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:16.242818 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:16.246322 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:16.246334 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:16.246355 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:16.246410 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:16.250133 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:16.250826 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:16.250837 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:16.250847 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:16.251060 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:16.251455 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:16.251463 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:16.251469 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:16.252105 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:16.252309 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:16.252981 (+ 672us) server_negotiation.cc:299] Negotiation successful
0924 19:03:16.253389 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":304,"threads_started":1}
I20260924 19:03:16.257236 14992 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45755
I20260924 19:03:16.257611 14992 heartbeater.cc:464] Registering TS with master...
I20260924 19:03:16.258297 14992 heartbeater.cc:511] Master 127.0.97.62:45755 requested a full tablet report, sending...
I20260924 19:03:16.260506 14807 ts_manager.cc:194] Registered new tserver with Master: fc0a539cb7784d15a1150b61ad525ce8 (127.0.97.1:39497)
I20260924 19:03:16.262076 14807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50853
W20260924 19:03:16.560489 14996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:03:16.560776 14996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:03:16.560969 14996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:03:16.567988 14996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:03:16.568121 14996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:03:16.568289 14996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:03:16.577667 14996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33037
--tserver_master_addrs=127.0.97.62:45755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:03:16.578872 14996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:03:16.580247 14996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:03:16.592686 15001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:16.593371 15002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:16.594549 15004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:03:16.595489 14996 server_base.cc:1034] running on GCE node
I20260924 19:03:16.596150 14996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:03:16.597047 14996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:03:16.598263 14996 hybrid_clock.cc:648] HybridClock initialized: now 1790276596598220 us; error 32 us; skew 500 ppm
I20260924 19:03:16.598570 14996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:03:16 dist-test-slave-9fn1 krb5kdc[14224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:03:16.601560 14996 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 19:03:16.603469 14996 webserver.cc:460] Webserver started at http://127.0.97.2:33037/ using document root <none> and password file <none>
I20260924 19:03:16.604415 14996 fs_manager.cc:359] Metadata directory not provided
I20260924 19:03:16.604565 14996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:03:16.611479 14996 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 19:03:16.614976 15011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:16.616285 14996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 19:03:16.616418 14996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "c55cf3a0245f4282a606a576da7f8c6d"
format_stamp: "Formatted at 2026-09-24 19:03:13 on dist-test-slave-9fn1"
server_key: "c65d999ab6ce624601c38519969218b5"
server_key_iv: "85a4ddea9b043f6b1820caec9b8aace5"
server_key_version: "encryptionkey@0"
I20260924 19:03:16.617358 14996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:03:16.632480 14996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:03:16.636639 14996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:03:16.636852 14996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:03:16.637765 14996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:03:16.638952 14996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:03:16.639046 14996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:16.639122 14996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:03:16.639179 14996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:16.671372 14996 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40787
I20260924 19:03:16.671422 15123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40787 every 8 connection(s)
I20260924 19:03:16.672889 14996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:03:16.675437 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 14996
I20260924 19:03:16.676141 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39891
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45755
--webserver_port=32927
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:03:16 dist-test-slave-9fn1 krb5kdc[14224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:03:16.694022 14994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:16.675383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38865 (local address 127.0.97.62:45755)
0924 19:03:16.675566 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:16.675575 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:16.676969 (+ 1394us) server_negotiation.cc:407] Connection header received
0924 19:03:16.678873 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:16.678879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:16.678970 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:16.679082 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:16.680541 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:16.681218 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:16.683392 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:16.683836 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:16.683936 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:16.687915 (+ 3979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:16.687919 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:16.687923 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:16.687953 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:16.690499 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:16.691238 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:16.691244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:16.691246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:16.691369 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:16.691885 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:16.691893 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:16.691899 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:16.692304 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:16.692483 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:16.693107 (+ 624us) server_negotiation.cc:299] Negotiation successful
0924 19:03:16.693387 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 19:03:16.694579 15126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:16.676014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.2:38865)
0924 19:03:16.676796 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:16.676837 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:16.678600 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:16.679322 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:16.679330 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:16.679817 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:03:16.680369 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:16.680386 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:16.681366 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:16.681372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:16.683215 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:16.683232 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:16.684057 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:16.684066 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:16.684388 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:16.685113 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:03:16.685144 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:03:16.687699 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:03:16.690661 (+ 2962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:16.690670 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:16.690685 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:16.691101 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:16.691482 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:16.691487 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:16.691491 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:16.691716 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:16.692440 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:03:16.692446 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:03:16.692732 (+ 286us) client_negotiation.cc:769] Sending connection context
0924 19:03:16.692940 (+ 208us) client_negotiation.cc:240] Negotiation successful
0924 19:03:16.693300 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":332,"threads_started":1}
I20260924 19:03:16.696055 15124 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45755
I20260924 19:03:16.696393 15124 heartbeater.cc:464] Registering TS with master...
I20260924 19:03:16.697310 15124 heartbeater.cc:511] Master 127.0.97.62:45755 requested a full tablet report, sending...
I20260924 19:03:16.698908 14807 ts_manager.cc:194] Registered new tserver with Master: c55cf3a0245f4282a606a576da7f8c6d (127.0.97.2:40787)
I20260924 19:03:16.699699 14807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38865
W20260924 19:03:17.001269 15127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:03:17.001595 15127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:03:17.001752 15127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:03:17.008860 15127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:03:17.008996 15127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:03:17.009158 15127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:03:17.018702 15127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=32927
--tserver_master_addrs=127.0.97.62:45755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:03:17.019903 15127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:03:17.021188 15127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:03:17.033964 15132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:17.033916 15135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:17.034008 15133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:03:17.034852 15127 server_base.cc:1034] running on GCE node
I20260924 19:03:17.035405 15127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:03:17.036283 15127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:03:17.037545 15127 hybrid_clock.cc:648] HybridClock initialized: now 1790276597037462 us; error 54 us; skew 500 ppm
I20260924 19:03:17.037868 15127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:03:17 dist-test-slave-9fn1 krb5kdc[14224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:03:17.040788 15127 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 19:03:17.042174 15127 webserver.cc:460] Webserver started at http://127.0.97.3:32927/ using document root <none> and password file <none>
I20260924 19:03:17.042793 15127 fs_manager.cc:359] Metadata directory not provided
I20260924 19:03:17.042953 15127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:03:17.047998 15127 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 19:03:17.050525 15142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:17.051734 15127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:03:17.051909 15127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "dabdbce8d75e497192b5effcced1c154"
format_stamp: "Formatted at 2026-09-24 19:03:14 on dist-test-slave-9fn1"
server_key: "0d80a45c3f85154140e6c12b762766f3"
server_key_iv: "c37e00dbb0cc7cd5261af5ab0988f023"
server_key_version: "encryptionkey@0"
I20260924 19:03:17.052857 15127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:03:17.071218 15127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:03:17.074950 15127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:03:17.075209 15127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:03:17.076131 15127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:03:17.077350 15127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:03:17.077477 15127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:17.077657 15127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:03:17.077747 15127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:17.110198 15127 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39891
I20260924 19:03:17.110281 15254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39891 every 8 connection(s)
I20260924 19:03:17.111575 15127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:03:17.118671 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 15127
Sep 24 19:03:17 dist-test-slave-9fn1 krb5kdc[14224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276597, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:03:17.132194 14994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:17.114052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39663 (local address 127.0.97.62:45755)
0924 19:03:17.114226 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:17.114230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:17.115175 (+ 945us) server_negotiation.cc:407] Connection header received
0924 19:03:17.116930 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:17.116935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:17.117036 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:17.117146 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:17.118610 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.119410 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.121448 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.122169 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.122302 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:17.125488 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:17.125494 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:17.125517 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:17.125552 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:17.128731 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:17.129705 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:17.129709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:17.129711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:17.129839 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:17.130361 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:17.130366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:17.130370 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:17.130680 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:17.130791 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:17.131672 (+ 881us) server_negotiation.cc:299] Negotiation successful
0924 19:03:17.131822 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 19:03:17.133358 15257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:17.114375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.3:39663)
0924 19:03:17.115026 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:17.115063 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:17.116672 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:17.117380 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:17.117389 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:17.117920 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:03:17.118448 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:17.118464 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.119609 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.119615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:17.121305 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:17.121317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.122371 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.122375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:17.122581 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:17.123246 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:03:17.123273 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:03:17.125311 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:03:17.128858 (+ 3547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:17.128867 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:17.128886 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:17.129583 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:17.129952 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:17.129956 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:17.129961 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:17.130219 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:17.130818 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:03:17.130826 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:03:17.131232 (+ 406us) client_negotiation.cc:769] Sending connection context
0924 19:03:17.131592 (+ 360us) client_negotiation.cc:240] Negotiation successful
0924 19:03:17.132075 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":255,"threads_started":1}
I20260924 19:03:17.134663 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:17.119576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44470 (local address 127.0.97.62:45755)
0924 19:03:17.120055 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:17.120059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:17.120479 (+ 420us) server_negotiation.cc:407] Connection header received
0924 19:03:17.120712 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:17.120716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:17.120764 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:17.120875 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:17.122609 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.123330 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.124591 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.125083 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.125222 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:17.128474 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:17.128479 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:17.128483 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:17.128523 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:17.131800 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:17.132588 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:17.132594 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:17.132597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:17.132732 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:17.133096 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:17.133101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:17.133104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:17.133435 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:17.133706 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:17.134076 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 19:03:17.134275 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":276,"threads_started":1}
I20260924 19:03:17.134903 15255 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45755
I20260924 19:03:17.135211 15255 heartbeater.cc:464] Registering TS with master...
I20260924 19:03:17.135900 15255 heartbeater.cc:511] Master 127.0.97.62:45755 requested a full tablet report, sending...
I20260924 19:03:17.137243 14807 ts_manager.cc:194] Registered new tserver with Master: dabdbce8d75e497192b5effcced1c154 (127.0.97.3:39891)
I20260924 19:03:17.138087 14807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:39663
I20260924 19:03:17.148778 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:03:17.155210 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:17.149608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44486 (local address 127.0.97.62:45755)
0924 19:03:17.149888 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:17.149897 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:17.150463 (+ 566us) server_negotiation.cc:407] Connection header received
0924 19:03:17.150804 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:17.150808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:17.150946 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:17.151060 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:03:17.151420 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.152069 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.153405 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.153851 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.153941 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:17.154355 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:03:17.154788 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:03:17.154898 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:17.154984 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 19:03:17.155039 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260924 19:03:17.156127 14807 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:44486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:03:17.163329 14807 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:03:17.184793 15268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:17.175367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41400 (local address 127.0.97.2:40787)
0924 19:03:17.176730 (+ 1363us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:17.176741 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:17.176775 (+ 34us) server_negotiation.cc:407] Connection header received
0924 19:03:17.176875 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:17.176885 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:17.177134 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:17.177347 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:03:17.178215 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.179938 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.182030 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.183548 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.183722 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:17.183935 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:17.184259 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 19:03:17.184447 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1207,"thread_start_us":1549,"threads_started":1}
I20260924 19:03:17.184821 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:17.174862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40787 (local address 127.0.0.1:41400)
0924 19:03:17.175694 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:17.175729 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:17.175906 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:17.177408 (+ 1502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:17.177419 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:17.177461 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:03:17.177965 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:17.177981 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.180115 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.180121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:17.181821 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:17.181833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.183819 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.183829 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:17.184002 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:17.184060 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 19:03:17.184205 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 19:03:17.184410 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":3,"spinlock_wait_cycles":8448,"thread_start_us":1194,"threads_started":1}
I20260924 19:03:17.186760 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:17.174714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37228 (local address 127.0.97.1:39497)
0924 19:03:17.179204 (+ 4490us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:17.179213 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:17.179239 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:03:17.179330 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:17.179338 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:17.179553 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:17.180276 (+ 723us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:03:17.180510 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.181976 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.184421 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.185717 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.185867 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:17.186064 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:17.186293 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 19:03:17.186439 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4335,"thread_start_us":4222,"threads_started":1}
I20260924 19:03:17.186843 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:17.174434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39497 (local address 127.0.0.1:37228)
0924 19:03:17.175688 (+ 1254us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:17.175729 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:17.175902 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:17.179782 (+ 3880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:17.179789 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:17.179818 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:03:17.180329 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:17.180343 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.182105 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.182113 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:17.184253 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:17.184265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.185922 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.185929 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:17.186095 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:17.186126 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 19:03:17.186243 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 19:03:17.186366 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"mutex_wait_us":145,"spinlock_wait_cycles":2944,"thread_start_us":1125,"threads_started":1}
I20260924 19:03:17.187249 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:17.176302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39891 (local address 127.0.0.1:58656)
0924 19:03:17.178565 (+ 2263us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:17.178579 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:17.178711 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:17.179286 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:17.179292 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:17.179318 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:03:17.179702 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:17.179720 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.181912 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.181920 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:17.184276 (+ 2356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:17.184289 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.186445 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.186452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:17.186643 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:17.186678 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 19:03:17.186746 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 19:03:17.186891 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2151,"mutex_wait_us":59,"thread_start_us":2036,"threads_started":1}
I20260924 19:03:17.188550 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:17.176671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58656 (local address 127.0.97.3:39891)
0924 19:03:17.178133 (+ 1462us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:17.178144 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:17.178672 (+ 528us) server_negotiation.cc:407] Connection header received
0924 19:03:17.178839 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:17.178849 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:17.179082 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:17.179905 (+ 823us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:03:17.180179 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.181778 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.184452 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.186206 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.187814 (+ 1608us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:17.188045 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:17.188107 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 19:03:17.188247 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1268,"thread_start_us":1130,"threads_started":1}
I20260924 19:03:17.188956 15059 tablet_service.cc:1467] Processing CreateTablet for tablet dfac65c61b0740b897d78a4933f6fca4 (DEFAULT_TABLE table=test-table [id=13588dd5bec54075b92d8a89b794e79b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:03:17.190558 15059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfac65c61b0740b897d78a4933f6fca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:03:17.191474 15190 tablet_service.cc:1467] Processing CreateTablet for tablet dfac65c61b0740b897d78a4933f6fca4 (DEFAULT_TABLE table=test-table [id=13588dd5bec54075b92d8a89b794e79b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:03:17.192032 14927 tablet_service.cc:1467] Processing CreateTablet for tablet dfac65c61b0740b897d78a4933f6fca4 (DEFAULT_TABLE table=test-table [id=13588dd5bec54075b92d8a89b794e79b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:03:17.192683 15190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfac65c61b0740b897d78a4933f6fca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:03:17.193222 14927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfac65c61b0740b897d78a4933f6fca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:03:17.204243 15271 tablet_bootstrap.cc:492] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154: Bootstrap starting.
I20260924 19:03:17.206538 15272 tablet_bootstrap.cc:492] T dfac65c61b0740b897d78a4933f6fca4 P c55cf3a0245f4282a606a576da7f8c6d: Bootstrap starting.
I20260924 19:03:17.208539 15271 tablet_bootstrap.cc:654] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154: Neither blocks nor log segments found. Creating new log.
I20260924 19:03:17.209779 15272 tablet_bootstrap.cc:654] T dfac65c61b0740b897d78a4933f6fca4 P c55cf3a0245f4282a606a576da7f8c6d: Neither blocks nor log segments found. Creating new log.
I20260924 19:03:17.209893 15271 log.cc:824] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154: Log is configured to *not* fsync() on all Append() calls
I20260924 19:03:17.210903 15272 log.cc:824] T dfac65c61b0740b897d78a4933f6fca4 P c55cf3a0245f4282a606a576da7f8c6d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:03:17.213461 15271 tablet_bootstrap.cc:492] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154: No bootstrap required, opened a new log
I20260924 19:03:17.213806 15271 ts_tablet_manager.cc:1397] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 19:03:17.214700 15273 tablet_bootstrap.cc:492] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8: Bootstrap starting.
I20260924 19:03:17.217587 15273 tablet_bootstrap.cc:654] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8: Neither blocks nor log segments found. Creating new log.
I20260924 19:03:17.218096 15272 tablet_bootstrap.cc:492] T dfac65c61b0740b897d78a4933f6fca4 P c55cf3a0245f4282a606a576da7f8c6d: No bootstrap required, opened a new log
I20260924 19:03:17.218389 15272 ts_tablet_manager.cc:1397] T dfac65c61b0740b897d78a4933f6fca4 P c55cf3a0245f4282a606a576da7f8c6d: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260924 19:03:17.218597 15273 log.cc:824] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8: Log is configured to *not* fsync() on all Append() calls
I20260924 19:03:17.221035 15273 tablet_bootstrap.cc:492] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8: No bootstrap required, opened a new log
I20260924 19:03:17.221261 15273 ts_tablet_manager.cc:1397] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 19:03:17.224426 15271 raft_consensus.cc:348] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc0a539cb7784d15a1150b61ad525ce8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39497 } } peers { permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 } } peers { permanent_uuid: "dabdbce8d75e497192b5effcced1c154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39891 } }
I20260924 19:03:17.224855 15271 raft_consensus.cc:374] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:03:17.224947 15271 raft_consensus.cc:729] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dabdbce8d75e497192b5effcced1c154, State: Initialized, Role: FOLLOWER
I20260924 19:03:17.225581 15271 consensus_queue.cc:260] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc0a539cb7784d15a1150b61ad525ce8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39497 } } peers { permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 } } peers { permanent_uuid: "dabdbce8d75e497192b5effcced1c154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39891 } }
I20260924 19:03:17.229421 15255 heartbeater.cc:503] Master 127.0.97.62:45755 was elected leader, sending a full tablet report...
I20260924 19:03:17.230134 15271 ts_tablet_manager.cc:1428] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260924 19:03:17.229954 15272 raft_consensus.cc:348] T dfac65c61b0740b897d78a4933f6fca4 P c55cf3a0245f4282a606a576da7f8c6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc0a539cb7784d15a1150b61ad525ce8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39497 } } peers { permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 } } peers { permanent_uuid: "dabdbce8d75e497192b5effcced1c154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39891 } }
I20260924 19:03:17.230381 15272 raft_consensus.cc:374] T dfac65c61b0740b897d78a4933f6fca4 P c55cf3a0245f4282a606a576da7f8c6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:03:17.230497 15272 raft_consensus.cc:729] T dfac65c61b0740b897d78a4933f6fca4 P c55cf3a0245f4282a606a576da7f8c6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c55cf3a0245f4282a606a576da7f8c6d, State: Initialized, Role: FOLLOWER
I20260924 19:03:17.231112 15273 raft_consensus.cc:348] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc0a539cb7784d15a1150b61ad525ce8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39497 } } peers { permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 } } peers { permanent_uuid: "dabdbce8d75e497192b5effcced1c154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39891 } }
I20260924 19:03:17.231540 15273 raft_consensus.cc:374] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:03:17.231637 15273 raft_consensus.cc:729] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc0a539cb7784d15a1150b61ad525ce8, State: Initialized, Role: FOLLOWER
I20260924 19:03:17.231652 15272 consensus_queue.cc:260] T dfac65c61b0740b897d78a4933f6fca4 P c55cf3a0245f4282a606a576da7f8c6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc0a539cb7784d15a1150b61ad525ce8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39497 } } peers { permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 } } peers { permanent_uuid: "dabdbce8d75e497192b5effcced1c154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39891 } }
I20260924 19:03:17.232297 15273 consensus_queue.cc:260] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc0a539cb7784d15a1150b61ad525ce8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39497 } } peers { permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 } } peers { permanent_uuid: "dabdbce8d75e497192b5effcced1c154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39891 } }
I20260924 19:03:17.234320 15124 heartbeater.cc:503] Master 127.0.97.62:45755 was elected leader, sending a full tablet report...
I20260924 19:03:17.236544 14992 heartbeater.cc:503] Master 127.0.97.62:45755 was elected leader, sending a full tablet report...
I20260924 19:03:17.236809 15272 ts_tablet_manager.cc:1428] T dfac65c61b0740b897d78a4933f6fca4 P c55cf3a0245f4282a606a576da7f8c6d: Time spent starting tablet: real 0.018s user 0.009s sys 0.005s
I20260924 19:03:17.237948 15273 ts_tablet_manager.cc:1428] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8: Time spent starting tablet: real 0.017s user 0.011s sys 0.004s
W20260924 19:03:17.363286 15256 tablet.cc:2369] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:03:17.424980 15125 tablet.cc:2369] T dfac65c61b0740b897d78a4933f6fca4 P c55cf3a0245f4282a606a576da7f8c6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:03:17.431228 15277 raft_consensus.cc:482] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:03:17.431475 15277 raft_consensus.cc:504] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc0a539cb7784d15a1150b61ad525ce8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39497 } } peers { permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 } } peers { permanent_uuid: "dabdbce8d75e497192b5effcced1c154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39891 } }
I20260924 19:03:17.433053 15277 leader_election.cc:288] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc0a539cb7784d15a1150b61ad525ce8 (127.0.97.1:39497), c55cf3a0245f4282a606a576da7f8c6d (127.0.97.2:40787)
I20260924 19:03:17.440200 15268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:17.433591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53217 (local address 127.0.97.2:40787)
0924 19:03:17.433824 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:17.433833 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:17.433922 (+ 89us) server_negotiation.cc:407] Connection header received
0924 19:03:17.434189 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:17.434195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:17.434272 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:17.434347 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:03:17.434842 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.435818 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.438211 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.439387 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.439515 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:17.439620 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:17.439886 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 19:03:17.439973 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 19:03:17.440308 15257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:17.433415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40787 (local address 127.0.97.3:53217)
0924 19:03:17.433805 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:17.433824 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:17.433998 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:17.434405 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:17.434409 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:17.434482 (+ 73us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:03:17.434667 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:17.434697 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.435963 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.435967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:17.438066 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:17.438081 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.439527 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.439531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:17.439664 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:17.439732 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 19:03:17.439842 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 19:03:17.439984 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":30,"spinlock_wait_cycles":22912}
I20260924 19:03:17.440537 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:17.433595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45493 (local address 127.0.97.1:39497)
0924 19:03:17.433807 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:17.433814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:17.434428 (+ 614us) server_negotiation.cc:407] Connection header received
0924 19:03:17.434574 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:17.434578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:17.434633 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:17.434702 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:03:17.435207 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.436127 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.438414 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.439613 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.439738 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:17.439828 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:17.440145 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 19:03:17.440294 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 19:03:17.440640 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:17.433459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39497 (local address 127.0.97.3:45493)
0924 19:03:17.434297 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:17.434309 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:17.434439 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:17.434743 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:17.434748 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:17.434775 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:03:17.435064 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:17.435071 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.436258 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.436264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:17.438279 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:17.438289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.439831 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.439837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:17.439991 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:17.440018 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 19:03:17.440101 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 19:03:17.440213 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"thread_start_us":473,"threads_started":1}
I20260924 19:03:17.441464 15079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfac65c61b0740b897d78a4933f6fca4" candidate_uuid: "dabdbce8d75e497192b5effcced1c154" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c55cf3a0245f4282a606a576da7f8c6d" is_pre_election: true
I20260924 19:03:17.441598 14947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfac65c61b0740b897d78a4933f6fca4" candidate_uuid: "dabdbce8d75e497192b5effcced1c154" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc0a539cb7784d15a1150b61ad525ce8" is_pre_election: true
I20260924 19:03:17.442008 15079 raft_consensus.cc:2454] T dfac65c61b0740b897d78a4933f6fca4 P c55cf3a0245f4282a606a576da7f8c6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dabdbce8d75e497192b5effcced1c154 in term 0.
I20260924 19:03:17.442098 14947 raft_consensus.cc:2454] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dabdbce8d75e497192b5effcced1c154 in term 0.
I20260924 19:03:17.442740 15145 leader_election.cc:302] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: c55cf3a0245f4282a606a576da7f8c6d, dabdbce8d75e497192b5effcced1c154, fc0a539cb7784d15a1150b61ad525ce8; no voters:
I20260924 19:03:17.443142 15277 raft_consensus.cc:2781] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:03:17.443300 15277 raft_consensus.cc:482] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:03:17.443399 15277 raft_consensus.cc:3037] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:03:17.445602 15277 raft_consensus.cc:504] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc0a539cb7784d15a1150b61ad525ce8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39497 } } peers { permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 } } peers { permanent_uuid: "dabdbce8d75e497192b5effcced1c154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39891 } }
I20260924 19:03:17.446183 15277 leader_election.cc:288] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154 [CANDIDATE]: Term 1 election: Requested vote from peers fc0a539cb7784d15a1150b61ad525ce8 (127.0.97.1:39497), c55cf3a0245f4282a606a576da7f8c6d (127.0.97.2:40787)
I20260924 19:03:17.446587 14947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfac65c61b0740b897d78a4933f6fca4" candidate_uuid: "dabdbce8d75e497192b5effcced1c154" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc0a539cb7784d15a1150b61ad525ce8"
I20260924 19:03:17.446609 15079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfac65c61b0740b897d78a4933f6fca4" candidate_uuid: "dabdbce8d75e497192b5effcced1c154" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c55cf3a0245f4282a606a576da7f8c6d"
I20260924 19:03:17.446776 14947 raft_consensus.cc:3037] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:03:17.446808 15079 raft_consensus.cc:3037] T dfac65c61b0740b897d78a4933f6fca4 P c55cf3a0245f4282a606a576da7f8c6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:03:17.448884 15079 raft_consensus.cc:2454] T dfac65c61b0740b897d78a4933f6fca4 P c55cf3a0245f4282a606a576da7f8c6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dabdbce8d75e497192b5effcced1c154 in term 1.
I20260924 19:03:17.448899 14947 raft_consensus.cc:2454] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dabdbce8d75e497192b5effcced1c154 in term 1.
I20260924 19:03:17.449270 15145 leader_election.cc:302] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c55cf3a0245f4282a606a576da7f8c6d, dabdbce8d75e497192b5effcced1c154; no voters:
I20260924 19:03:17.449582 15277 raft_consensus.cc:2781] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:03:17.450006 15277 raft_consensus.cc:686] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154 [term 1 LEADER]: Becoming Leader. State: Replica: dabdbce8d75e497192b5effcced1c154, State: Running, Role: LEADER
I20260924 19:03:17.450443 15277 consensus_queue.cc:237] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc0a539cb7784d15a1150b61ad525ce8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39497 } } peers { permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 } } peers { permanent_uuid: "dabdbce8d75e497192b5effcced1c154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39891 } }
I20260924 19:03:17.454550 14806 catalog_manager.cc:5495] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154 reported cstate change: term changed from 0 to 1, leader changed from <none> to dabdbce8d75e497192b5effcced1c154 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "dabdbce8d75e497192b5effcced1c154" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc0a539cb7784d15a1150b61ad525ce8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dabdbce8d75e497192b5effcced1c154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39891 } health_report { overall_health: HEALTHY } } }
I20260924 19:03:17.478631 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:17.473842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58662 (local address 127.0.97.3:39891)
0924 19:03:17.474000 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:17.474004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:17.474024 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:03:17.474195 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:17.474198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:17.474265 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:17.474381 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:03:17.474679 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.475377 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.476815 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.477191 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.477280 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:17.477744 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:03:17.478203 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:03:17.478306 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:17.478376 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 19:03:17.478432 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 19:03:17.482908 14993 tablet.cc:2369] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:03:17.486763 14947 raft_consensus.cc:1261] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [term 1 FOLLOWER]: Refusing update from remote peer dabdbce8d75e497192b5effcced1c154: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:03:17.486773 15079 raft_consensus.cc:1261] T dfac65c61b0740b897d78a4933f6fca4 P c55cf3a0245f4282a606a576da7f8c6d [term 1 FOLLOWER]: Refusing update from remote peer dabdbce8d75e497192b5effcced1c154: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:03:17.487797 15281 consensus_queue.cc:1035] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc0a539cb7784d15a1150b61ad525ce8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:03:17.488152 15277 consensus_queue.cc:1035] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:03:17.512643 14807 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:44486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:03:17.512926 14807 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:44486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:03:17.516278 14807 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b: Sending DeleteTablet for 3 replicas of tablet dfac65c61b0740b897d78a4933f6fca4
I20260924 19:03:17.517483 15059 tablet_service.cc:1514] Processing DeleteTablet for tablet dfac65c61b0740b897d78a4933f6fca4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:03:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41400
I20260924 19:03:17.517717 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 14844
I20260924 19:03:17.517776 15190 tablet_service.cc:1514] Processing DeleteTablet for tablet dfac65c61b0740b897d78a4933f6fca4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:03:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58656
I20260924 19:03:17.518285 14927 tablet_service.cc:1514] Processing DeleteTablet for tablet dfac65c61b0740b897d78a4933f6fca4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:03:17 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37228
I20260924 19:03:17.530445 15295 tablet_replica.cc:330] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154: stopping tablet replica
I20260924 19:03:17.531044 15295 raft_consensus.cc:2229] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:03:17.531778 15295 raft_consensus.cc:2258] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:03:17.534102 15295 ts_tablet_manager.cc:1905] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:03:17.541414 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:17.519391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37234 (local address 127.0.97.1:39497)
0924 19:03:17.519591 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:17.519599 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:17.519619 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:03:17.519681 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:17.519689 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:17.519845 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:17.519952 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:17.522295 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.523678 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.526106 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.527046 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.527193 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:17.531545 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:17.531564 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:17.531588 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:17.531636 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:17.536487 (+ 4851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:17.537577 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:17.537589 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:17.537594 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:17.537777 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:17.538286 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:17.538294 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:17.538300 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:17.538882 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:17.539203 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:17.540639 (+ 1436us) server_negotiation.cc:299] Negotiation successful
0924 19:03:17.540916 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 19:03:17.542277 14987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:03:17.541438 15295 ts_tablet_manager.cc:1918] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:03:17.542541 15295 log.cc:1196] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/wal/wals/dfac65c61b0740b897d78a4933f6fca4
I20260924 19:03:17.542894 15294 tablet_replica.cc:330] T dfac65c61b0740b897d78a4933f6fca4 P c55cf3a0245f4282a606a576da7f8c6d: stopping tablet replica
I20260924 19:03:17.543344 15294 raft_consensus.cc:2229] T dfac65c61b0740b897d78a4933f6fca4 P c55cf3a0245f4282a606a576da7f8c6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:03:17.543630 15295 ts_tablet_manager.cc:1939] T dfac65c61b0740b897d78a4933f6fca4 P dabdbce8d75e497192b5effcced1c154: Deleting consensus metadata
I20260924 19:03:17.543967 15294 raft_consensus.cc:2258] T dfac65c61b0740b897d78a4933f6fca4 P c55cf3a0245f4282a606a576da7f8c6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:03:17.546336 15294 ts_tablet_manager.cc:1905] T dfac65c61b0740b897d78a4933f6fca4 P c55cf3a0245f4282a606a576da7f8c6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:03:17.546602 14792 catalog_manager.cc:4841] TS dabdbce8d75e497192b5effcced1c154 (127.0.97.3:39891): tablet dfac65c61b0740b897d78a4933f6fca4 (table test-table [id=13588dd5bec54075b92d8a89b794e79b]) successfully deleted
I20260924 19:03:17.553566 15294 ts_tablet_manager.cc:1918] T dfac65c61b0740b897d78a4933f6fca4 P c55cf3a0245f4282a606a576da7f8c6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:03:17.553741 15294 log.cc:1196] T dfac65c61b0740b897d78a4933f6fca4 P c55cf3a0245f4282a606a576da7f8c6d: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/wal/wals/dfac65c61b0740b897d78a4933f6fca4
I20260924 19:03:17.554266 15294 ts_tablet_manager.cc:1939] T dfac65c61b0740b897d78a4933f6fca4 P c55cf3a0245f4282a606a576da7f8c6d: Deleting consensus metadata
I20260924 19:03:17.556037 14793 catalog_manager.cc:4841] TS c55cf3a0245f4282a606a576da7f8c6d (127.0.97.2:40787): tablet dfac65c61b0740b897d78a4933f6fca4 (table test-table [id=13588dd5bec54075b92d8a89b794e79b]) successfully deleted
I20260924 19:03:17.635697 15296 tablet_replica.cc:330] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8: stopping tablet replica
I20260924 19:03:17.636032 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 14844
W20260924 19:03:17.653853 14792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:39497 (error 108)
I20260924 19:03:17.654260 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 14996
I20260924 19:03:17.656016 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:17.655457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39497 (local address 127.0.0.1:37236)
0924 19:03:17.655734 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:17.655850 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
W20260924 19:03:17.656335 14792 catalog_manager.cc:4584] TS fc0a539cb7784d15a1150b61ad525ce8 (127.0.97.1:39497): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dfac65c61b0740b897d78a4933f6fca4: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39497: connect: Connection refused (error 111)
I20260924 19:03:17.668604 15268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:17.655172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41410 (local address 127.0.97.2:40787)
0924 19:03:17.655359 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:17.655367 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:17.655386 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:03:17.655748 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:17.655752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:17.655869 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:17.655992 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:17.657581 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.658305 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.659677 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.660089 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.660234 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:17.662943 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:17.662956 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:17.662970 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:17.663001 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:17.665913 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:17.666601 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:17.666608 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:17.666611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:17.666741 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:17.667133 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:17.667139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:17.667142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:17.667492 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:17.667776 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:17.668069 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 19:03:17.668288 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:03:17.669412 15119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:03:17.710119 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:17.709581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39497 (local address 127.0.0.1:37238)
0924 19:03:17.709818 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:17.709947 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 19:03:17.711272 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:17.710852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39497 (local address 127.0.0.1:37244)
0924 19:03:17.711017 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:17.711133 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 19:03:17.760172 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 14996
I20260924 19:03:17.771912 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:17.771372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39497 (local address 127.0.0.1:37260)
0924 19:03:17.771608 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:17.771721 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 19:03:17.773115 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:17.772690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39497 (local address 127.0.0.1:37264)
0924 19:03:17.772865 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:17.772970 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 19:03:17.775861 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 15127
I20260924 19:03:17.794616 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:17.780932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58676 (local address 127.0.97.3:39891)
0924 19:03:17.781118 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:17.781126 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:17.781148 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:03:17.781369 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:17.781391 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:17.781471 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:17.781840 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:17.783132 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.783954 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.785310 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:17.785867 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:17.786004 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:17.788584 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:17.788595 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:17.788610 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:17.788647 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:17.791670 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:17.792400 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:17.792407 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:17.792409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:17.792553 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:17.792939 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:17.792945 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:17.792948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:17.793293 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:17.793645 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:17.793952 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 19:03:17.794171 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:03:17.795341 15250 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:03:17.853995 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:17.853456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39497 (local address 127.0.0.1:37266)
0924 19:03:17.853727 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:17.853822 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260924 19:03:17.855072 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:17.854706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39497 (local address 127.0.0.1:37278)
0924 19:03:17.854898 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:17.854972 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260924 19:03:17.916808 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 15127
I20260924 19:03:17.936517 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 14774
I20260924 19:03:17.937675 14837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:03:18.050945 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:18.050297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39497 (local address 127.0.0.1:37292)
0924 19:03:18.050533 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:18.050657 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:39497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 19:03:18.051131 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 14774
I20260924 19:03:18.072471 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45755
--webserver_interface=127.0.97.62
--webserver_port=35637
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:03:18.406852 15302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:03:18.407261 15302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:03:18.407444 15302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:03:18.417369 15302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:03:18.417553 15302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:03:18.417640 15302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:03:18.417722 15302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:03:18.417788 15302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:03:18.427917 15302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=35637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:03:18.429332 15302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:03:18.430745 15302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:03:18.442444 15310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:18.442384 15307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:18.442823 15308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:03:18.443289 15302 server_base.cc:1034] running on GCE node
I20260924 19:03:18.443964 15302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:03:18.445250 15302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:03:18.446445 15302 hybrid_clock.cc:648] HybridClock initialized: now 1790276598446365 us; error 66 us; skew 500 ppm
I20260924 19:03:18.446746 15302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:03:18 dist-test-slave-9fn1 krb5kdc[14224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:03:18.449889 15302 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:03:18.451270 15302 webserver.cc:460] Webserver started at http://127.0.97.62:35637/ using document root <none> and password file <none>
I20260924 19:03:18.451956 15302 fs_manager.cc:359] Metadata directory not provided
I20260924 19:03:18.452087 15302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:03:18.457358 15302 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 19:03:18.460088 15317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:18.461390 15302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:03:18.461583 15302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "8e301c22a72249bc9d55d0eda728c70b"
format_stamp: "Formatted at 2026-09-24 19:03:12 on dist-test-slave-9fn1"
server_key: "23db884e266ad7796146c591501f6a3d"
server_key_iv: "8a931e9f2fff36850c1a5ad7ee20c339"
server_key_version: "encryptionkey@0"
I20260924 19:03:18.462538 15302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:03:18.477432 15302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:03:18.481355 15302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:03:18.481614 15302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:03:18.499495 15302 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45755
I20260924 19:03:18.499518 15368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45755 every 8 connection(s)
I20260924 19:03:18.500998 15302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:03:18.503546 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 15302
I20260924 19:03:18.504370 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39497
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:45755
--webserver_port=33025
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:03:18.507896 15369 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:03:18.513996 15369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b: Bootstrap starting.
I20260924 19:03:18.545545 15369 log.cc:824] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b: Log is configured to *not* fsync() on all Append() calls
I20260924 19:03:18.562886 15369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:03:18.563443 15369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b: Bootstrap complete.
I20260924 19:03:18.572781 15369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e301c22a72249bc9d55d0eda728c70b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45755 } }
I20260924 19:03:18.573736 15369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e301c22a72249bc9d55d0eda728c70b, State: Initialized, Role: FOLLOWER
I20260924 19:03:18.574442 15369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e301c22a72249bc9d55d0eda728c70b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45755 } }
I20260924 19:03:18.574673 15369 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:03:18.574786 15369 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:03:18.574987 15369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [term 2 FOLLOWER]: Advancing to term 3
I20260924 19:03:18.578737 15369 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e301c22a72249bc9d55d0eda728c70b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45755 } }
I20260924 19:03:18.579229 15369 leader_election.cc:302] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e301c22a72249bc9d55d0eda728c70b; no voters:
I20260924 19:03:18.580160 15369 leader_election.cc:288] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 19:03:18.580554 15373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [term 3 FOLLOWER]: Leader election won for term 3
I20260924 19:03:18.587386 15373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [term 3 LEADER]: Becoming Leader. State: Replica: 8e301c22a72249bc9d55d0eda728c70b, State: Running, Role: LEADER
I20260924 19:03:18.587963 15373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e301c22a72249bc9d55d0eda728c70b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45755 } }
I20260924 19:03:18.588759 15369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:03:18.590588 15375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e301c22a72249bc9d55d0eda728c70b. Latest consensus state: current_term: 3 leader_uuid: "8e301c22a72249bc9d55d0eda728c70b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e301c22a72249bc9d55d0eda728c70b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45755 } } }
I20260924 19:03:18.590629 15374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8e301c22a72249bc9d55d0eda728c70b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e301c22a72249bc9d55d0eda728c70b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45755 } } }
I20260924 19:03:18.590970 15375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [sys.catalog]: This master's current role is: LEADER
I20260924 19:03:18.591028 15374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e301c22a72249bc9d55d0eda728c70b [sys.catalog]: This master's current role is: LEADER
I20260924 19:03:18.603655 15378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:03:18.614408 15378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:03:18.616269 15378 catalog_manager.cc:1259] Loaded cluster ID: ab8033854edd473da4e1ed318474d999
I20260924 19:03:18.616392 15378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 19:03:18.618966 15378 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbaf16599ad google::LogMessage::Fail() at ??:0
@ 0x7fbaf165c88a google::LogMessage::SendToLog() at ??:0
@ 0x7fbaf16594de google::LogMessage::Flush() at ??:0
@ 0x7fbaf165aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbb0708c512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fbb07089976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbb07111610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fbb07111439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fbaf42a5345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbaf42a891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fbaf42a8799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fbaf4283eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbaf755d6db start_thread at ??:0
@ 0x7fbaf04f361f clone at ??:0
W20260924 19:03:18.905761 15371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:03:18.906088 15371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:03:18.906260 15371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:03:18.913699 15371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:03:18.913856 15371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:03:18.914031 15371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:03:18.924362 15371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=33025
--tserver_master_addrs=127.0.97.62:45755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:03:18.925737 15371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:03:18.927029 15371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:03:18.939795 15411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:18.939898 15410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:18.939819 15413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:03:18.946147 15371 server_base.cc:1034] running on GCE node
I20260924 19:03:18.946842 15371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:03:18.947769 15371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:03:18.948966 15371 hybrid_clock.cc:648] HybridClock initialized: now 1790276598948905 us; error 52 us; skew 500 ppm
I20260924 19:03:18.949272 15371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:03:18 dist-test-slave-9fn1 krb5kdc[14224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:03:18.952406 15371 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 19:03:18.953959 15371 webserver.cc:460] Webserver started at http://127.0.97.1:33025/ using document root <none> and password file <none>
I20260924 19:03:18.954625 15371 fs_manager.cc:359] Metadata directory not provided
I20260924 19:03:18.954725 15371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:03:18.959843 15371 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 19:03:18.962697 15420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:18.963960 15371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:03:18.964115 15371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "fc0a539cb7784d15a1150b61ad525ce8"
format_stamp: "Formatted at 2026-09-24 19:03:13 on dist-test-slave-9fn1"
server_key: "81deae84fae9525c71a42bb55270b7ce"
server_key_iv: "0e0de2429461d95bf0dab6a086a14d59"
server_key_version: "encryptionkey@0"
I20260924 19:03:18.965035 15371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:03:18.975982 15371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:03:18.980399 15371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:03:18.980726 15371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:03:18.981765 15371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:03:18.983971 15427 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260924 19:03:18.989454 15371 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 19:03:18.989609 15371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.001s
I20260924 19:03:18.989735 15371 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 19:03:18.992902 15371 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 19:03:18.992980 15371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260924 19:03:18.993345 15427 tablet_bootstrap.cc:492] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8: Bootstrap starting.
I20260924 19:03:19.020147 15427 log.cc:824] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8: Log is configured to *not* fsync() on all Append() calls
I20260924 19:03:19.034641 15371 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39497
I20260924 19:03:19.034694 15534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39497 every 8 connection(s)
I20260924 19:03:19.036113 15371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:03:19.037169 15427 tablet_bootstrap.cc:492] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:03:19.038043 15427 tablet_bootstrap.cc:492] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8: Bootstrap complete.
I20260924 19:03:19.039754 15427 ts_tablet_manager.cc:1397] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8: Time spent bootstrapping tablet: real 0.047s user 0.025s sys 0.018s
I20260924 19:03:19.041049 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:19.039048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.1:45117)
0924 19:03:19.039918 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:19.040167 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":382,"threads_started":1}
W20260924 19:03:19.041857 15535 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45755 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45755: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
I20260924 19:03:19.042150 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 15371
I20260924 19:03:19.042845 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:19.042435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.1:32875)
0924 19:03:19.042609 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:19.042701 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 19:03:19.042857 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:40787
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:45755
--webserver_port=33037
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:03:19.044122 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:19.043686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.1:34971)
0924 19:03:19.043852 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:19.043954 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 19:03:19.044430 15535 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 19:03:19.047636 15427 raft_consensus.cc:348] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc0a539cb7784d15a1150b61ad525ce8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39497 } } peers { permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 } } peers { permanent_uuid: "dabdbce8d75e497192b5effcced1c154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39891 } }
I20260924 19:03:19.048501 15427 raft_consensus.cc:729] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc0a539cb7784d15a1150b61ad525ce8, State: Initialized, Role: FOLLOWER
I20260924 19:03:19.049162 15427 consensus_queue.cc:260] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc0a539cb7784d15a1150b61ad525ce8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39497 } } peers { permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 } } peers { permanent_uuid: "dabdbce8d75e497192b5effcced1c154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39891 } }
I20260924 19:03:19.050714 15427 ts_tablet_manager.cc:1428] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8: Time spent starting tablet: real 0.011s user 0.000s sys 0.012s
I20260924 19:03:19.050794 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:19.050384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.1:58469)
0924 19:03:19.050547 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:19.050632 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 19:03:19.051643 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:19.051360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.1:56037)
0924 19:03:19.051482 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:19.051540 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 19:03:19.361171 15538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:03:19.361485 15538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:03:19.361681 15538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:03:19.368868 15538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:03:19.369012 15538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:03:19.369194 15538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:03:19.378515 15538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:40787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=33037
--tserver_master_addrs=127.0.97.62:45755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:03:19.379807 15538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:03:19.381072 15538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:03:19.393803 15545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:19.393883 15547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:19.394354 15544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:03:19.394841 15538 server_base.cc:1034] running on GCE node
I20260924 19:03:19.395435 15538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:03:19.396344 15538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:03:19.397631 15538 hybrid_clock.cc:648] HybridClock initialized: now 1790276599397532 us; error 75 us; skew 500 ppm
I20260924 19:03:19.397964 15538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:03:19 dist-test-slave-9fn1 krb5kdc[14224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:03:19.401158 15538 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 19:03:19.402546 15538 webserver.cc:460] Webserver started at http://127.0.97.2:33037/ using document root <none> and password file <none>
I20260924 19:03:19.403199 15538 fs_manager.cc:359] Metadata directory not provided
I20260924 19:03:19.403298 15538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:03:19.408579 15538 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 19:03:19.412818 15554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:19.414273 15538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:03:19.414409 15538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "c55cf3a0245f4282a606a576da7f8c6d"
format_stamp: "Formatted at 2026-09-24 19:03:13 on dist-test-slave-9fn1"
server_key: "c65d999ab6ce624601c38519969218b5"
server_key_iv: "85a4ddea9b043f6b1820caec9b8aace5"
server_key_version: "encryptionkey@0"
I20260924 19:03:19.415354 15538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:03:19.432519 15538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:03:19.436884 15538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:03:19.437114 15538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:03:19.438102 15538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:03:19.439373 15538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:03:19.439460 15538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:19.439553 15538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:03:19.439605 15538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:19.472648 15538 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40787
I20260924 19:03:19.472707 15666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40787 every 8 connection(s)
I20260924 19:03:19.474208 15538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:03:19.475203 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 15538
I20260924 19:03:19.475962 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39891
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:45755
--webserver_port=32927
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:03:19.479786 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:19.477475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.2:54229)
0924 19:03:19.478356 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:19.478602 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":294,"threads_started":1}
W20260924 19:03:19.480440 15667 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45755 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45755: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
I20260924 19:03:19.481357 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:19.480935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.2:36125)
0924 19:03:19.481117 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:19.481206 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 19:03:19.482278 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:19.481868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.2:53493)
0924 19:03:19.482035 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:19.482131 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 19:03:19.482517 15667 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:03:19.797367 15670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:03:19.797766 15670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:03:19.797946 15670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:03:19.804998 15670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:03:19.805136 15670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:03:19.805323 15670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:03:19.814908 15670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=32927
--tserver_master_addrs=127.0.97.62:45755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:03:19.816251 15670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:03:19.817821 15670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:03:19.830999 15678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:19.831046 15676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:19.831076 15675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:03:19.831909 15670 server_base.cc:1034] running on GCE node
I20260924 19:03:19.832525 15670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:03:19.833398 15670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:03:19.834610 15670 hybrid_clock.cc:648] HybridClock initialized: now 1790276599834576 us; error 65 us; skew 500 ppm
I20260924 19:03:19.834913 15670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:03:19 dist-test-slave-9fn1 krb5kdc[14224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276599, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:03:19.837946 15670 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 19:03:19.839298 15670 webserver.cc:460] Webserver started at http://127.0.97.3:32927/ using document root <none> and password file <none>
I20260924 19:03:19.839956 15670 fs_manager.cc:359] Metadata directory not provided
I20260924 19:03:19.840106 15670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:03:19.845091 15670 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:03:19.847656 15685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:19.848896 15670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:03:19.849067 15670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "dabdbce8d75e497192b5effcced1c154"
format_stamp: "Formatted at 2026-09-24 19:03:14 on dist-test-slave-9fn1"
server_key: "0d80a45c3f85154140e6c12b762766f3"
server_key_iv: "c37e00dbb0cc7cd5261af5ab0988f023"
server_key_version: "encryptionkey@0"
I20260924 19:03:19.850057 15670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:03:19.879676 15670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:03:19.883961 15670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:03:19.884253 15670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:03:19.885212 15670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:03:19.886529 15670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:03:19.886659 15670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:19.886792 15670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:03:19.886880 15670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:19.919195 15670 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39891
I20260924 19:03:19.919257 15797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39891 every 8 connection(s)
I20260924 19:03:19.920770 15670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:03:19.925403 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:19.923682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.3:57861)
0924 19:03:19.924371 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:19.924577 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":264,"threads_started":1}
W20260924 19:03:19.926059 15798 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:45755 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:45755: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
I20260924 19:03:19.926854 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:19.926535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.3:43891)
0924 19:03:19.926683 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:19.926756 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 19:03:19.927623 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:19.927322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.3:34633)
0924 19:03:19.927452 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:19.927528 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 19:03:19.927841 15798 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 19:03:19.928263 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 15670
I20260924 19:03:20.053184 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:20.052380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.1:54333)
0924 19:03:20.052851 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:20.052943 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":249,"threads_started":1}
I20260924 19:03:20.245208 15803 raft_consensus.cc:482] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:03:20.245536 15803 raft_consensus.cc:504] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc0a539cb7784d15a1150b61ad525ce8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39497 } } peers { permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 } } peers { permanent_uuid: "dabdbce8d75e497192b5effcced1c154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39891 } }
I20260924 19:03:20.247190 15803 leader_election.cc:288] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c55cf3a0245f4282a606a576da7f8c6d (127.0.97.2:40787), dabdbce8d75e497192b5effcced1c154 (127.0.97.3:39891)
Sep 24 19:03:20 dist-test-slave-9fn1 krb5kdc[14224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
Sep 24 19:03:20 dist-test-slave-9fn1 krb5kdc[14224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 19:03:20.270431 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:20.247622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40787 (local address 127.0.97.1:57903)
0924 19:03:20.249641 (+ 2019us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:20.249701 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:20.250382 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:20.253292 (+ 2910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:20.253298 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:20.254180 (+ 882us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:03:20.254907 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:20.254924 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:20.256445 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:20.256455 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:20.257925 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:20.257936 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:20.258926 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:20.258933 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:20.259204 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:20.260124 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:03:20.260148 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:03:20.263063 (+ 2915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:03:20.266710 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:20.266713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:20.266715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:20.267191 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:20.267633 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:20.267636 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:20.267638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:20.267763 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:20.268629 (+ 866us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:03:20.268637 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:03:20.269014 (+ 377us) client_negotiation.cc:769] Sending connection context
0924 19:03:20.269298 (+ 284us) client_negotiation.cc:240] Negotiation successful
0924 19:03:20.269916 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1841,"mutex_wait_us":323,"spinlock_wait_cycles":35072}
I20260924 19:03:20.270475 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:20.247892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39823 (local address 127.0.97.3:39891)
0924 19:03:20.250144 (+ 2252us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:20.250161 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:20.250582 (+ 421us) server_negotiation.cc:407] Connection header received
0924 19:03:20.252359 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:20.252368 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:20.252832 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:20.253178 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:20.255117 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:20.256252 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:20.258060 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:20.258691 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:20.258883 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:20.262982 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:20.262993 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:20.263028 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:20.263081 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:20.266339 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:20.267193 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:20.267201 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:20.267208 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:20.267385 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:20.267873 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:20.267877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:20.267883 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:20.268512 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:20.268704 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:20.269644 (+ 940us) server_negotiation.cc:299] Negotiation successful
0924 19:03:20.270074 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2093,"thread_start_us":1943,"threads_started":1}
I20260924 19:03:20.270442 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:20.247834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39891 (local address 127.0.97.1:39823)
0924 19:03:20.250462 (+ 2628us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:20.250481 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:20.250621 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:20.253239 (+ 2618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:20.253249 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:20.254218 (+ 969us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:03:20.254905 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:20.254923 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:20.256446 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:20.256449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:20.257918 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:20.257929 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:20.258904 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:20.258911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:20.259209 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:20.260216 (+ 1007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:03:20.260232 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:03:20.262683 (+ 2451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:03:20.266478 (+ 3795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:20.266487 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:20.266502 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:20.267047 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:20.267499 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:20.267503 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:20.267506 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:20.267747 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:20.268758 (+ 1011us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:03:20.268762 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:03:20.268991 (+ 229us) client_negotiation.cc:769] Sending connection context
0924 19:03:20.269298 (+ 307us) client_negotiation.cc:240] Negotiation successful
0924 19:03:20.270028 (+ 730us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2534,"mutex_wait_us":387,"spinlock_wait_cycles":2048,"thread_start_us":2395,"threads_started":1}
I20260924 19:03:20.271623 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:20.247715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57903 (local address 127.0.97.2:40787)
0924 19:03:20.249599 (+ 1884us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:20.249623 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:20.250101 (+ 478us) server_negotiation.cc:407] Connection header received
0924 19:03:20.252379 (+ 2278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:20.252387 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:20.252772 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:20.253104 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:20.255102 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:20.256265 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:20.258062 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:20.258690 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:20.258894 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:20.263322 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:20.263328 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:20.263339 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:20.263374 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:20.266555 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:20.267320 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:20.267326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:20.267331 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:20.267517 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:20.267925 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:20.267933 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:20.267938 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:20.268470 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:20.268684 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:20.269655 (+ 971us) server_negotiation.cc:299] Negotiation successful
0924 19:03:20.270070 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1620,"thread_start_us":1477,"threads_started":1}
W20260924 19:03:20.274852 15422 leader_election.cc:341] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dabdbce8d75e497192b5effcced1c154 (127.0.97.3:39891): Not found: Tablet not found: dfac65c61b0740b897d78a4933f6fca4
W20260924 19:03:20.275104 15423 leader_election.cc:341] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c55cf3a0245f4282a606a576da7f8c6d (127.0.97.2:40787): Not found: Tablet not found: dfac65c61b0740b897d78a4933f6fca4
I20260924 19:03:20.275244 15423 leader_election.cc:302] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc0a539cb7784d15a1150b61ad525ce8; no voters: c55cf3a0245f4282a606a576da7f8c6d, dabdbce8d75e497192b5effcced1c154
I20260924 19:03:20.275681 15803 raft_consensus.cc:2726] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:03:20.483794 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:20.482977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.2:58327)
0924 19:03:20.483413 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:20.483558 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":216,"threads_started":1}
I20260924 19:03:20.929018 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:20.928330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.3:51843)
0924 19:03:20.928728 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:20.928825 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":207,"threads_started":1}
I20260924 19:03:21.054802 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:21.054088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.1:48345)
0924 19:03:21.054573 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:21.054647 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":247,"threads_started":1}
I20260924 19:03:21.485234 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:21.484607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.2:41941)
0924 19:03:21.485011 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:21.485086 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":211,"threads_started":1}
I20260924 19:03:21.930795 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:21.930052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.3:58819)
0924 19:03:21.930534 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:21.930615 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":275,"threads_started":1}
I20260924 19:03:22.056435 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:22.055606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.1:41749)
0924 19:03:22.056146 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:22.056262 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":228,"threads_started":1}
I20260924 19:03:22.344936 15813 raft_consensus.cc:482] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:03:22.345182 15813 raft_consensus.cc:504] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc0a539cb7784d15a1150b61ad525ce8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39497 } } peers { permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 } } peers { permanent_uuid: "dabdbce8d75e497192b5effcced1c154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39891 } }
I20260924 19:03:22.345934 15813 leader_election.cc:288] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c55cf3a0245f4282a606a576da7f8c6d (127.0.97.2:40787), dabdbce8d75e497192b5effcced1c154 (127.0.97.3:39891)
W20260924 19:03:22.346969 15422 leader_election.cc:341] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dabdbce8d75e497192b5effcced1c154 (127.0.97.3:39891): Not found: Tablet not found: dfac65c61b0740b897d78a4933f6fca4
W20260924 19:03:22.347177 15423 leader_election.cc:341] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c55cf3a0245f4282a606a576da7f8c6d (127.0.97.2:40787): Not found: Tablet not found: dfac65c61b0740b897d78a4933f6fca4
I20260924 19:03:22.347312 15423 leader_election.cc:302] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc0a539cb7784d15a1150b61ad525ce8; no voters: c55cf3a0245f4282a606a576da7f8c6d, dabdbce8d75e497192b5effcced1c154
I20260924 19:03:22.347585 15813 raft_consensus.cc:2726] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:03:22.486727 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:22.486074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.2:43107)
0924 19:03:22.486497 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:22.486573 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":222,"threads_started":1}
I20260924 19:03:22.932479 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:22.931814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.3:49539)
0924 19:03:22.932244 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:22.932316 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":225,"threads_started":1}
I20260924 19:03:23.058184 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:23.057339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.1:45971)
0924 19:03:23.057899 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:23.057985 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":315,"threads_started":1}
I20260924 19:03:23.488304 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:23.487626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.2:50085)
0924 19:03:23.488033 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:23.488108 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":196,"threads_started":1}
I20260924 19:03:23.934063 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:23.933349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.3:50935)
0924 19:03:23.933818 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:23.933898 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":258,"threads_started":1}
I20260924 19:03:24.059847 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:24.059202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.1:34605)
0924 19:03:24.059596 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:24.059669 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":205,"threads_started":1}
I20260924 19:03:24.490180 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:24.489372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.2:53111)
0924 19:03:24.489898 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:24.490011 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":323,"threads_started":1}
I20260924 19:03:24.633337 15821 raft_consensus.cc:482] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:03:24.633565 15821 raft_consensus.cc:504] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc0a539cb7784d15a1150b61ad525ce8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39497 } } peers { permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 } } peers { permanent_uuid: "dabdbce8d75e497192b5effcced1c154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39891 } }
I20260924 19:03:24.634236 15821 leader_election.cc:288] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c55cf3a0245f4282a606a576da7f8c6d (127.0.97.2:40787), dabdbce8d75e497192b5effcced1c154 (127.0.97.3:39891)
W20260924 19:03:24.635210 15422 leader_election.cc:341] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dabdbce8d75e497192b5effcced1c154 (127.0.97.3:39891): Not found: Tablet not found: dfac65c61b0740b897d78a4933f6fca4
W20260924 19:03:24.635423 15423 leader_election.cc:341] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c55cf3a0245f4282a606a576da7f8c6d (127.0.97.2:40787): Not found: Tablet not found: dfac65c61b0740b897d78a4933f6fca4
I20260924 19:03:24.635576 15423 leader_election.cc:302] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc0a539cb7784d15a1150b61ad525ce8; no voters: c55cf3a0245f4282a606a576da7f8c6d, dabdbce8d75e497192b5effcced1c154
I20260924 19:03:24.635810 15821 raft_consensus.cc:2726] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:03:24.935647 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:24.934970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.3:54849)
0924 19:03:24.935414 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:24.935497 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":241,"threads_started":1}
I20260924 19:03:25.061650 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:25.060745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.1:48243)
0924 19:03:25.061264 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:25.061383 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":271,"threads_started":1}
I20260924 19:03:25.491830 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:25.491148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.2:58487)
0924 19:03:25.491586 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:25.491674 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":250,"threads_started":1}
I20260924 19:03:25.937371 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:25.936654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.3:32899)
0924 19:03:25.937110 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:25.937191 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":239,"threads_started":1}
I20260924 19:03:26.063306 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:26.062620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.1:52537)
0924 19:03:26.063034 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:26.063112 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":223,"threads_started":1}
I20260924 19:03:26.493408 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:26.492659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.2:49359)
0924 19:03:26.493139 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:26.493226 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":213,"threads_started":1}
I20260924 19:03:26.939069 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:26.938299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.3:38449)
0924 19:03:26.938791 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:26.938866 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":216,"threads_started":1}
I20260924 19:03:27.064826 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:27.064178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.1:48599)
0924 19:03:27.064582 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:27.064656 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":212,"threads_started":1}
I20260924 19:03:27.312486 15830 raft_consensus.cc:482] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:03:27.312660 15830 raft_consensus.cc:504] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc0a539cb7784d15a1150b61ad525ce8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39497 } } peers { permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 } } peers { permanent_uuid: "dabdbce8d75e497192b5effcced1c154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39891 } }
I20260924 19:03:27.313239 15830 leader_election.cc:288] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c55cf3a0245f4282a606a576da7f8c6d (127.0.97.2:40787), dabdbce8d75e497192b5effcced1c154 (127.0.97.3:39891)
W20260924 19:03:27.314285 15422 leader_election.cc:341] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dabdbce8d75e497192b5effcced1c154 (127.0.97.3:39891): Not found: Tablet not found: dfac65c61b0740b897d78a4933f6fca4
W20260924 19:03:27.314464 15423 leader_election.cc:341] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c55cf3a0245f4282a606a576da7f8c6d (127.0.97.2:40787): Not found: Tablet not found: dfac65c61b0740b897d78a4933f6fca4
I20260924 19:03:27.314563 15423 leader_election.cc:302] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc0a539cb7784d15a1150b61ad525ce8; no voters: c55cf3a0245f4282a606a576da7f8c6d, dabdbce8d75e497192b5effcced1c154
I20260924 19:03:27.314774 15830 raft_consensus.cc:2726] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:03:27.494969 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:27.494321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.2:48085)
0924 19:03:27.494709 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:27.494787 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":201,"threads_started":1}
I20260924 19:03:27.940562 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:27.939881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.3:60309)
0924 19:03:27.940317 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:27.940395 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":234,"threads_started":1}
I20260924 19:03:28.066521 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:28.065813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.1:42195)
0924 19:03:28.066257 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:28.066348 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":224,"threads_started":1}
I20260924 19:03:28.496516 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:28.495859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.2:40831)
0924 19:03:28.496298 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:28.496373 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":228,"threads_started":1}
I20260924 19:03:28.942169 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:28.941404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.3:42017)
0924 19:03:28.941917 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:28.942014 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":273,"threads_started":1}
I20260924 19:03:29.068150 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:29.067443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.1:46289)
0924 19:03:29.067868 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:29.067971 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":224,"threads_started":1}
I20260924 19:03:29.498216 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:29.497477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.2:33543)
0924 19:03:29.497967 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:29.498044 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":267,"threads_started":1}
I20260924 19:03:29.943866 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:29.943085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.3:54637)
0924 19:03:29.943597 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:29.943682 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":274,"threads_started":1}
I20260924 19:03:30.069831 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:30.068990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.1:43115)
0924 19:03:30.069483 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:30.069595 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":272,"threads_started":1}
I20260924 19:03:30.499814 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:30.499103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.2:42367)
0924 19:03:30.499569 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:30.499650 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":231,"threads_started":1}
I20260924 19:03:30.945600 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:30.944799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.3:39239)
0924 19:03:30.945287 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:30.945379 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":255,"threads_started":1}
I20260924 19:03:31.071228 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:31.070571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.1:56259)
0924 19:03:31.070981 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:31.071054 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":234,"threads_started":1}
I20260924 19:03:31.136828 15843 raft_consensus.cc:482] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:03:31.137024 15843 raft_consensus.cc:504] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc0a539cb7784d15a1150b61ad525ce8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39497 } } peers { permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 } } peers { permanent_uuid: "dabdbce8d75e497192b5effcced1c154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39891 } }
I20260924 19:03:31.137738 15843 leader_election.cc:288] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c55cf3a0245f4282a606a576da7f8c6d (127.0.97.2:40787), dabdbce8d75e497192b5effcced1c154 (127.0.97.3:39891)
W20260924 19:03:31.138688 15422 leader_election.cc:341] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dabdbce8d75e497192b5effcced1c154 (127.0.97.3:39891): Not found: Tablet not found: dfac65c61b0740b897d78a4933f6fca4
W20260924 19:03:31.138895 15423 leader_election.cc:341] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c55cf3a0245f4282a606a576da7f8c6d (127.0.97.2:40787): Not found: Tablet not found: dfac65c61b0740b897d78a4933f6fca4
I20260924 19:03:31.139005 15423 leader_election.cc:302] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc0a539cb7784d15a1150b61ad525ce8; no voters: c55cf3a0245f4282a606a576da7f8c6d, dabdbce8d75e497192b5effcced1c154
I20260924 19:03:31.139279 15843 raft_consensus.cc:2726] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:03:31.501747 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:31.500934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.2:37725)
0924 19:03:31.501417 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:31.501560 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":250,"threads_started":1}
I20260924 19:03:31.947487 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:31.946627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.3:42699)
0924 19:03:31.947225 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:31.947304 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":346,"threads_started":1}
I20260924 19:03:32.072997 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:32.072254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.1:51693)
0924 19:03:32.072696 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:32.072789 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":241,"threads_started":1}
I20260924 19:03:32.503307 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:32.502625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.2:58691)
0924 19:03:32.503056 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:32.503141 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":217,"threads_started":1}
I20260924 19:03:32.948894 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:32.948265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.3:38419)
0924 19:03:32.948663 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:32.948735 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":200,"threads_started":1}
I20260924 19:03:32.974391 15849 raft_consensus.cc:482] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:03:32.974613 15849 raft_consensus.cc:504] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc0a539cb7784d15a1150b61ad525ce8" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39497 } } peers { permanent_uuid: "c55cf3a0245f4282a606a576da7f8c6d" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40787 } } peers { permanent_uuid: "dabdbce8d75e497192b5effcced1c154" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39891 } }
I20260924 19:03:32.975291 15849 leader_election.cc:288] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c55cf3a0245f4282a606a576da7f8c6d (127.0.97.2:40787), dabdbce8d75e497192b5effcced1c154 (127.0.97.3:39891)
W20260924 19:03:32.976222 15422 leader_election.cc:341] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dabdbce8d75e497192b5effcced1c154 (127.0.97.3:39891): Not found: Tablet not found: dfac65c61b0740b897d78a4933f6fca4
W20260924 19:03:32.976397 15423 leader_election.cc:341] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c55cf3a0245f4282a606a576da7f8c6d (127.0.97.2:40787): Not found: Tablet not found: dfac65c61b0740b897d78a4933f6fca4
I20260924 19:03:32.976491 15423 leader_election.cc:302] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc0a539cb7784d15a1150b61ad525ce8; no voters: c55cf3a0245f4282a606a576da7f8c6d, dabdbce8d75e497192b5effcced1c154
I20260924 19:03:32.976727 15849 raft_consensus.cc:2726] T dfac65c61b0740b897d78a4933f6fca4 P fc0a539cb7784d15a1150b61ad525ce8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:03:33.074489 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:33.073804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.1:56035)
0924 19:03:33.074223 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:33.074307 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":227,"threads_started":1}
I20260924 19:03:33.505059 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:33.504296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.2:39871)
0924 19:03:33.504770 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:33.504852 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":228,"threads_started":1}
I20260924 19:03:33.950598 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:33.949820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.3:57099)
0924 19:03:33.950290 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:33.950372 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":255,"threads_started":1}
I20260924 19:03:34.076247 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:34.075593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.1:58013)
0924 19:03:34.075993 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:34.076079 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":202,"threads_started":1}
I20260924 19:03:34.507126 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:34.506372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.2:39927)
0924 19:03:34.506863 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:34.506965 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":230,"threads_started":1}
I20260924 19:03:34.936340 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 15371
I20260924 19:03:34.952260 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:34.951631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45755 (local address 127.0.97.3:44169)
0924 19:03:34.952013 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:34.952088 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:45755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":195,"threads_started":1}
I20260924 19:03:34.968573 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:34.937289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56202 (local address 127.0.97.1:39497)
0924 19:03:34.937953 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:34.937963 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:34.937993 (+ 30us) server_negotiation.cc:407] Connection header received
0924 19:03:34.938063 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:34.938069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:34.938273 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:34.938464 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:34.940696 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:34.941762 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:34.943027 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:34.943570 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:34.943709 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:34.962836 (+ 19127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:34.962851 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:34.962866 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:34.962904 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:34.965790 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:34.966470 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:34.966478 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:34.966480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:34.966608 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:34.967005 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:34.967009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:34.967011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:34.967368 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:34.967670 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:34.967999 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 19:03:34.968262 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":275,"threads_started":1}
I20260924 19:03:34.969960 15530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:03:35.061709 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 15371
I20260924 19:03:35.079707 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 15538
I20260924 19:03:35.092006 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:35.080458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47018 (local address 127.0.97.2:40787)
0924 19:03:35.080870 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:35.080874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:35.080890 (+ 16us) server_negotiation.cc:407] Connection header received
0924 19:03:35.080943 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:35.080946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:35.081006 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:35.081098 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:35.082378 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:35.083160 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:35.084284 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:35.084678 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:35.084765 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:35.087331 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:35.087336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:35.087340 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:35.087369 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:35.089871 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:35.090398 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:35.090401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:35.090403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:35.090528 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:35.090818 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:35.090821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:35.090823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:35.091064 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:35.091252 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:35.091464 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 19:03:35.091628 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":214,"threads_started":1}
I20260924 19:03:35.092662 15662 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:03:35.176158 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 15538
I20260924 19:03:35.193892 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 15670
I20260924 19:03:35.207247 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:35.194682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44430 (local address 127.0.97.3:39891)
0924 19:03:35.195231 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:35.195235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:35.195252 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:03:35.195305 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:35.195308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:35.195367 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:35.195492 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:35.197033 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:35.197906 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:35.199177 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:35.199604 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:35.199732 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:35.202252 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:35.202257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:35.202260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:35.202291 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:35.204627 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:35.205265 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:35.205270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:35.205274 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:35.205399 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:35.205808 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:35.205813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:35.205817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:35.206087 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:35.206381 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:35.206645 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 19:03:35.206875 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":278,"threads_started":1}
I20260924 19:03:35.207940 15793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:03:35.290431 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 15670
2026-09-24T19:03:35Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24110 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:03:35 dist-test-slave-9fn1 krb5kdc[15864](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:03:35 dist-test-slave-9fn1 krb5kdc[15864](info): set up 2 sockets
Sep 24 19:03:35 dist-test-slave-9fn1 krb5kdc[15864](info): commencing operation
krb5kdc: starting...
W20260924 19:03:36.377985 388 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.012s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:03:36 dist-test-slave-9fn1 krb5kdc[15864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:03:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:03:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 19:03:36.612490 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42697
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:03:36.943815 15877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:03:36.944074 15877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:03:36.944181 15877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:03:36.951234 15877 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 19:03:36.951319 15877 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 19:03:36.951356 15877 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 19:03:36.951398 15877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:03:36.951500 15877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:03:36.951617 15877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:03:36.951692 15877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:03:36.951779 15877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:03:36.961158 15877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:03:36.962397 15877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:03:36.963685 15877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:03:36.975672 15883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:36.975708 15885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:36.975724 15882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:03:36.976112 15877 server_base.cc:1034] running on GCE node
I20260924 19:03:36.976868 15877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:03:36.978255 15877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:03:36.979485 15877 hybrid_clock.cc:648] HybridClock initialized: now 1790276616979429 us; error 26 us; skew 500 ppm
I20260924 19:03:36.979848 15877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:03:36 dist-test-slave-9fn1 krb5kdc[15864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276616, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:03:36.982815 15877 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:03:36.984083 15877 webserver.cc:460] Webserver started at http://127.0.97.62:36985/ using document root <none> and password file <none>
I20260924 19:03:36.984687 15877 fs_manager.cc:359] Metadata directory not provided
I20260924 19:03:36.984767 15877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:03:36.985015 15877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:03:36.987825 15877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "276693ed963e44a6827ffc8d97d8356f"
format_stamp: "Formatted at 2026-09-24 19:03:36 on dist-test-slave-9fn1"
server_key: "a8c1ce2fe86d58c058774f59c89fa6fd"
server_key_iv: "55309b4e33838f818092f9bca0d7d690"
server_key_version: "encryptionkey@0"
I20260924 19:03:36.988507 15877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "276693ed963e44a6827ffc8d97d8356f"
format_stamp: "Formatted at 2026-09-24 19:03:36 on dist-test-slave-9fn1"
server_key: "a8c1ce2fe86d58c058774f59c89fa6fd"
server_key_iv: "55309b4e33838f818092f9bca0d7d690"
server_key_version: "encryptionkey@0"
I20260924 19:03:36.993387 15877 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 19:03:36.996747 15892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:36.998131 15877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:03:36.998266 15877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "276693ed963e44a6827ffc8d97d8356f"
format_stamp: "Formatted at 2026-09-24 19:03:36 on dist-test-slave-9fn1"
server_key: "a8c1ce2fe86d58c058774f59c89fa6fd"
server_key_iv: "55309b4e33838f818092f9bca0d7d690"
server_key_version: "encryptionkey@0"
I20260924 19:03:36.998440 15877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:03:37.013244 15877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:03:37.017042 15877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:03:37.017246 15877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:03:37.035579 15877 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42697
I20260924 19:03:37.035655 15943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42697 every 8 connection(s)
I20260924 19:03:37.036931 15877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:03:37.040493 15944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:03:37.043999 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 15877
I20260924 19:03:37.044212 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:03:37.044883 388 external_mini_cluster.cc:1428] Setting key 82ebe405c24772ea725d6573e2b58cd7
I20260924 19:03:37.051157 15944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f: Bootstrap starting.
I20260924 19:03:37.054869 15944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f: Neither blocks nor log segments found. Creating new log.
I20260924 19:03:37.056022 15944 log.cc:824] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f: Log is configured to *not* fsync() on all Append() calls
I20260924 19:03:37.058862 15944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f: No bootstrap required, opened a new log
Sep 24 19:03:37 dist-test-slave-9fn1 krb5kdc[15864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:03:37.069537 15947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:37.046717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53516 (local address 127.0.97.62:42697)
0924 19:03:37.047477 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:37.047488 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:37.047527 (+ 39us) server_negotiation.cc:407] Connection header received
0924 19:03:37.049724 (+ 2197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:37.049763 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:37.050121 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:37.050426 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:37.052367 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:37.053731 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:37.055650 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:37.056682 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:37.056896 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:37.060906 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:37.060918 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:37.060937 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:37.060983 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:37.064997 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:37.065856 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:37.065864 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:37.065872 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:37.066082 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:37.066426 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:37.066431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:37.066435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:37.067148 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:37.067452 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:37.067783 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 19:03:37.068214 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":367,"threads_started":1}
I20260924 19:03:37.069290 15944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276693ed963e44a6827ffc8d97d8356f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42697 } }
I20260924 19:03:37.069743 15944 raft_consensus.cc:374] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:03:37.069895 15944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 276693ed963e44a6827ffc8d97d8356f, State: Initialized, Role: FOLLOWER
I20260924 19:03:37.070479 15944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276693ed963e44a6827ffc8d97d8356f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42697 } }
I20260924 19:03:37.070710 15944 raft_consensus.cc:388] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:03:37.070873 15944 raft_consensus.cc:482] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:03:37.071058 15944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:03:37.073289 15944 raft_consensus.cc:504] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276693ed963e44a6827ffc8d97d8356f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42697 } }
I20260924 19:03:37.073798 15944 leader_election.cc:302] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 276693ed963e44a6827ffc8d97d8356f; no voters:
I20260924 19:03:37.074375 15944 leader_election.cc:288] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:03:37.074507 15949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:03:37.075227 15949 raft_consensus.cc:686] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [term 1 LEADER]: Becoming Leader. State: Replica: 276693ed963e44a6827ffc8d97d8356f, State: Running, Role: LEADER
I20260924 19:03:37.075759 15949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276693ed963e44a6827ffc8d97d8356f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42697 } }
I20260924 19:03:37.076385 15944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:03:37.078640 15950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "276693ed963e44a6827ffc8d97d8356f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276693ed963e44a6827ffc8d97d8356f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42697 } } }
I20260924 19:03:37.078814 15951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 276693ed963e44a6827ffc8d97d8356f. Latest consensus state: current_term: 1 leader_uuid: "276693ed963e44a6827ffc8d97d8356f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276693ed963e44a6827ffc8d97d8356f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42697 } } }
I20260924 19:03:37.079082 15950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [sys.catalog]: This master's current role is: LEADER
I20260924 19:03:37.079103 15951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [sys.catalog]: This master's current role is: LEADER
I20260924 19:03:37.080761 15954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:03:37.086552 15954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:03:37.095196 15954 catalog_manager.cc:1347] Generated new cluster ID: 71d271b6a53a4395bc084281617a1158
I20260924 19:03:37.095388 15954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:03:37.115811 15954 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:03:37.117136 15954 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:03:37.124001 15954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f: Generated new TSK 0
I20260924 19:03:37.124761 15954 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 19:03:37.184664 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42697
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:03:37.509238 15970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:03:37.509482 15970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:03:37.509637 15970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:03:37.516628 15970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:03:37.516728 15970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:03:37.516835 15970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:03:37.526536 15970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:03:37.527747 15970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:03:37.528976 15970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:03:37.542219 15975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:37.542382 15978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:37.542513 15976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:03:37.543431 15970 server_base.cc:1034] running on GCE node
I20260924 19:03:37.544001 15970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:03:37.545013 15970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:03:37.546253 15970 hybrid_clock.cc:648] HybridClock initialized: now 1790276617546197 us; error 44 us; skew 500 ppm
I20260924 19:03:37.546552 15970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:03:37 dist-test-slave-9fn1 krb5kdc[15864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:03:37.549571 15970 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 19:03:37.550925 15970 webserver.cc:460] Webserver started at http://127.0.97.1:42257/ using document root <none> and password file <none>
I20260924 19:03:37.551586 15970 fs_manager.cc:359] Metadata directory not provided
I20260924 19:03:37.551728 15970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:03:37.552048 15970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:03:37.554919 15970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/data/instance:
uuid: "5c688e4544644d3aa51249bdd95e0485"
format_stamp: "Formatted at 2026-09-24 19:03:37 on dist-test-slave-9fn1"
server_key: "b7a4105efd9ae35f9d7a5fc0d3e0f65b"
server_key_iv: "2b8b963ddd332d35098011b4d0a70d3c"
server_key_version: "encryptionkey@0"
I20260924 19:03:37.555689 15970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5c688e4544644d3aa51249bdd95e0485"
format_stamp: "Formatted at 2026-09-24 19:03:37 on dist-test-slave-9fn1"
server_key: "b7a4105efd9ae35f9d7a5fc0d3e0f65b"
server_key_iv: "2b8b963ddd332d35098011b4d0a70d3c"
server_key_version: "encryptionkey@0"
I20260924 19:03:37.560467 15970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260924 19:03:37.563686 15985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:37.564895 15970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 19:03:37.565069 15970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "5c688e4544644d3aa51249bdd95e0485"
format_stamp: "Formatted at 2026-09-24 19:03:37 on dist-test-slave-9fn1"
server_key: "b7a4105efd9ae35f9d7a5fc0d3e0f65b"
server_key_iv: "2b8b963ddd332d35098011b4d0a70d3c"
server_key_version: "encryptionkey@0"
I20260924 19:03:37.565286 15970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:03:37.577458 15970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:03:37.581624 15970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:03:37.581894 15970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:03:37.582818 15970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:03:37.584074 15970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:03:37.584173 15970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:37.584313 15970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:03:37.584384 15970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:37.616917 15970 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44949
I20260924 19:03:37.616972 16097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44949 every 8 connection(s)
I20260924 19:03:37.618258 15970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:03:37.626559 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 15970
I20260924 19:03:37.626766 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/wal/instance
I20260924 19:03:37.627431 388 external_mini_cluster.cc:1428] Setting key 9d8e3a74d7b0c975b75075eaf9cadc71
Sep 24 19:03:37 dist-test-slave-9fn1 krb5kdc[15864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276617, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:03:37.638671 16100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:37.620747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48133 (local address 127.0.97.62:42697)
0924 19:03:37.621170 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:37.621176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:37.622184 (+ 1008us) server_negotiation.cc:407] Connection header received
0924 19:03:37.624037 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:37.624084 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:37.624174 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:37.624331 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:37.625771 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:37.626567 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:37.628378 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:37.628995 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:37.629121 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:37.633060 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:37.633065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:37.633068 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:37.633097 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:37.635672 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:37.636380 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:37.636390 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:37.636392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:37.636504 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:37.636919 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:37.636924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:37.636928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:37.637182 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:37.637336 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:37.638078 (+ 742us) server_negotiation.cc:299] Negotiation successful
0924 19:03:37.638295 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":194,"threads_started":1}
I20260924 19:03:37.639578 16101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:37.621316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.1:48133)
0924 19:03:37.622039 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:37.622075 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:37.623787 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:37.624509 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:37.624516 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:37.625002 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:03:37.625615 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:37.625643 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:37.626759 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:37.626763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:37.628189 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:37.628206 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:37.629245 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:37.629256 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:37.629549 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:37.630261 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:03:37.630304 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:03:37.632852 (+ 2548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:03:37.635813 (+ 2961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:37.635821 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:37.635837 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:37.636253 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:37.636610 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:37.636616 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:37.636620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:37.636803 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:37.637306 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:03:37.637313 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:03:37.637712 (+ 399us) client_negotiation.cc:769] Sending connection context
0924 19:03:37.637968 (+ 256us) client_negotiation.cc:240] Negotiation successful
0924 19:03:37.638482 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":308,"threads_started":1}
I20260924 19:03:37.640987 16098 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42697
I20260924 19:03:37.641312 16098 heartbeater.cc:464] Registering TS with master...
I20260924 19:03:37.642207 16098 heartbeater.cc:511] Master 127.0.97.62:42697 requested a full tablet report, sending...
I20260924 19:03:37.644330 15909 ts_manager.cc:194] Registered new tserver with Master: 5c688e4544644d3aa51249bdd95e0485 (127.0.97.1:44949)
I20260924 19:03:37.645730 15909 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48133
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 19:03:37.678562 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42697
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:03:37.993732 16104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:03:37.993981 16104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:03:37.994071 16104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:03:38.001029 16104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:03:38.001129 16104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:03:38.001245 16104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:03:38.010905 16104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:03:38.012001 16104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:03:38.013175 16104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:03:38.025681 16109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:38.025712 16110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:38.025781 16112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:03:38.026865 16104 server_base.cc:1034] running on GCE node
I20260924 19:03:38.027433 16104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:03:38.028301 16104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:03:38.029575 16104 hybrid_clock.cc:648] HybridClock initialized: now 1790276618029471 us; error 46 us; skew 500 ppm
I20260924 19:03:38.029901 16104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:03:38 dist-test-slave-9fn1 krb5kdc[15864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:03:38.032804 16104 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 19:03:38.034228 16104 webserver.cc:460] Webserver started at http://127.0.97.2:43137/ using document root <none> and password file <none>
I20260924 19:03:38.034861 16104 fs_manager.cc:359] Metadata directory not provided
I20260924 19:03:38.035017 16104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:03:38.035326 16104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:03:38.038539 16104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/data/instance:
uuid: "54662ec87ba34e70a719d89f450b8d1e"
format_stamp: "Formatted at 2026-09-24 19:03:38 on dist-test-slave-9fn1"
server_key: "c4099ecfadc935f586bd0166104dd6e1"
server_key_iv: "0972755de12e7748cde063fc253c9e6f"
server_key_version: "encryptionkey@0"
I20260924 19:03:38.039263 16104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/wal/instance:
uuid: "54662ec87ba34e70a719d89f450b8d1e"
format_stamp: "Formatted at 2026-09-24 19:03:38 on dist-test-slave-9fn1"
server_key: "c4099ecfadc935f586bd0166104dd6e1"
server_key_iv: "0972755de12e7748cde063fc253c9e6f"
server_key_version: "encryptionkey@0"
I20260924 19:03:38.044239 16104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:03:38.047655 16119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:38.049039 16104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:03:38.049193 16104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "54662ec87ba34e70a719d89f450b8d1e"
format_stamp: "Formatted at 2026-09-24 19:03:38 on dist-test-slave-9fn1"
server_key: "c4099ecfadc935f586bd0166104dd6e1"
server_key_iv: "0972755de12e7748cde063fc253c9e6f"
server_key_version: "encryptionkey@0"
I20260924 19:03:38.049399 16104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:03:38.061033 16104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:03:38.064770 16104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:03:38.065040 16104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:03:38.066057 16104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:03:38.067358 16104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:03:38.067433 16104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:38.067593 16104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:03:38.067658 16104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:38.100615 16104 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41921
I20260924 19:03:38.100725 16231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41921 every 8 connection(s)
I20260924 19:03:38.102077 16104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:03:38.108551 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 16104
I20260924 19:03:38.108762 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/wal/instance
I20260924 19:03:38.109354 388 external_mini_cluster.cc:1428] Setting key ee23b4e587e31fdfac972b4c3a67fccb
Sep 24 19:03:38 dist-test-slave-9fn1 krb5kdc[15864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:03:38.123401 16100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:38.104572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60701 (local address 127.0.97.62:42697)
0924 19:03:38.104771 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:38.104775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:38.105888 (+ 1113us) server_negotiation.cc:407] Connection header received
0924 19:03:38.107714 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:38.107720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:38.107811 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:38.107921 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:38.110028 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.110863 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.113111 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.113611 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.113708 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:38.117765 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:38.117770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:38.117774 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:38.117805 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:38.120390 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:38.121111 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:38.121116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:38.121119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:38.121253 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:38.121756 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:38.121760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:38.121762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:38.122013 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:38.122186 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:38.122835 (+ 649us) server_negotiation.cc:299] Negotiation successful
0924 19:03:38.123047 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 19:03:38.124284 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:38.105001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.2:60701)
0924 19:03:38.105735 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:38.105777 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:38.107455 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:38.108184 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:38.108196 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:38.109000 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:03:38.109801 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:38.109826 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.111034 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.111039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:38.112951 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:38.112969 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.113822 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.113831 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:38.114152 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:38.114890 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:03:38.114920 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:03:38.117492 (+ 2572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:03:38.120533 (+ 3041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:38.120541 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:38.120557 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:38.120977 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:38.121365 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:38.121370 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:38.121374 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:38.121622 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:38.122147 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:03:38.122154 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:03:38.122491 (+ 337us) client_negotiation.cc:769] Sending connection context
0924 19:03:38.122744 (+ 253us) client_negotiation.cc:240] Negotiation successful
0924 19:03:38.123225 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":280,"threads_started":1}
I20260924 19:03:38.125738 16232 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42697
I20260924 19:03:38.126045 16232 heartbeater.cc:464] Registering TS with master...
I20260924 19:03:38.126821 16232 heartbeater.cc:511] Master 127.0.97.62:42697 requested a full tablet report, sending...
I20260924 19:03:38.128353 15909 ts_manager.cc:194] Registered new tserver with Master: 54662ec87ba34e70a719d89f450b8d1e (127.0.97.2:41921)
I20260924 19:03:38.129125 15909 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60701
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 19:03:38.162810 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42697
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:03:38.488401 16237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:03:38.488828 16237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:03:38.489035 16237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:03:38.500510 16237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:03:38.500649 16237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:03:38.500849 16237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:03:38.513229 16237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:03:38.514647 16237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:03:38.515903 16237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:03:38.528975 16243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:38.529021 16245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:38.529592 16242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:03:38.529914 16237 server_base.cc:1034] running on GCE node
I20260924 19:03:38.530572 16237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:03:38.531478 16237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:03:38.532718 16237 hybrid_clock.cc:648] HybridClock initialized: now 1790276618532672 us; error 57 us; skew 500 ppm
I20260924 19:03:38.533034 16237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:03:38 dist-test-slave-9fn1 krb5kdc[15864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:03:38.536160 16237 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 19:03:38.537463 16237 webserver.cc:460] Webserver started at http://127.0.97.3:43851/ using document root <none> and password file <none>
I20260924 19:03:38.538174 16237 fs_manager.cc:359] Metadata directory not provided
I20260924 19:03:38.538273 16237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:03:38.538523 16237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:03:38.541229 16237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/data/instance:
uuid: "4db698cbb4b045b98841a62ac7a61b49"
format_stamp: "Formatted at 2026-09-24 19:03:38 on dist-test-slave-9fn1"
server_key: "3e71cc6f3d136d932d619dd1ec3b42bc"
server_key_iv: "1817b158ce5782877eade30f2c36cf4c"
server_key_version: "encryptionkey@0"
I20260924 19:03:38.541983 16237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4db698cbb4b045b98841a62ac7a61b49"
format_stamp: "Formatted at 2026-09-24 19:03:38 on dist-test-slave-9fn1"
server_key: "3e71cc6f3d136d932d619dd1ec3b42bc"
server_key_iv: "1817b158ce5782877eade30f2c36cf4c"
server_key_version: "encryptionkey@0"
I20260924 19:03:38.546710 16237 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 19:03:38.550024 16252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:38.551285 16237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:03:38.551409 16237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "4db698cbb4b045b98841a62ac7a61b49"
format_stamp: "Formatted at 2026-09-24 19:03:38 on dist-test-slave-9fn1"
server_key: "3e71cc6f3d136d932d619dd1ec3b42bc"
server_key_iv: "1817b158ce5782877eade30f2c36cf4c"
server_key_version: "encryptionkey@0"
I20260924 19:03:38.551568 16237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:03:38.581832 16237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:03:38.587018 16237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:03:38.587304 16237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:03:38.588217 16237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:03:38.589469 16237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:03:38.589622 16237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:38.589773 16237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:03:38.589824 16237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:38.623414 16237 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44109
I20260924 19:03:38.623497 16364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44109 every 8 connection(s)
I20260924 19:03:38.624740 16237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:03:38.634312 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 16237
I20260924 19:03:38.634507 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/wal/instance
I20260924 19:03:38.635012 388 external_mini_cluster.cc:1428] Setting key 145be645173947b9074bb7fbc6116896
Sep 24 19:03:38 dist-test-slave-9fn1 krb5kdc[15864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:03:38.644330 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:38.627271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57851 (local address 127.0.97.62:42697)
0924 19:03:38.627735 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:38.627740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:38.628513 (+ 773us) server_negotiation.cc:407] Connection header received
0924 19:03:38.630378 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:38.630384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:38.630482 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:38.630597 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:38.631996 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.632814 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.634674 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.635080 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.635237 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:38.639164 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:38.639168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:38.639172 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:38.639204 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:38.641855 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:38.642459 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:38.642463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:38.642465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:38.642571 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:38.642958 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:38.642961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:38.642963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:38.643209 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:38.643313 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:38.643883 (+ 570us) server_negotiation.cc:299] Negotiation successful
0924 19:03:38.644033 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":273,"threads_started":1}
I20260924 19:03:38.645157 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:38.627699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.3:57851)
0924 19:03:38.628349 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:38.628384 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:38.630126 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:38.630819 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:38.630827 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:38.631316 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:03:38.631833 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:38.631849 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.632971 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.632975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:38.634507 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:38.634524 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.635347 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.635356 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:38.635627 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:38.636378 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:03:38.636409 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:03:38.638981 (+ 2572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:03:38.641958 (+ 2977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:38.641964 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:38.641974 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:38.642366 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:38.642672 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:03:38.642675 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:03:38.642678 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:03:38.642871 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:03:38.643302 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:03:38.643307 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:03:38.643591 (+ 284us) client_negotiation.cc:769] Sending connection context
0924 19:03:38.643790 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 19:03:38.644123 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":266,"threads_started":1}
I20260924 19:03:38.646562 16365 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42697
I20260924 19:03:38.646854 16365 heartbeater.cc:464] Registering TS with master...
I20260924 19:03:38.647500 16365 heartbeater.cc:511] Master 127.0.97.62:42697 requested a full tablet report, sending...
I20260924 19:03:38.648784 15909 ts_manager.cc:194] Registered new tserver with Master: 4db698cbb4b045b98841a62ac7a61b49 (127.0.97.3:44109)
I20260924 19:03:38.649350 16098 heartbeater.cc:503] Master 127.0.97.62:42697 was elected leader, sending a full tablet report...
I20260924 19:03:38.649838 15909 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57851
I20260924 19:03:38.650404 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:03:38.665848 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:38.654778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53526 (local address 127.0.97.62:42697)
0924 19:03:38.654930 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:38.654934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:38.655361 (+ 427us) server_negotiation.cc:407] Connection header received
0924 19:03:38.655483 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:38.655486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:38.655557 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:38.655642 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:38.656949 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.657683 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.658943 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.659343 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.659433 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:38.661526 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:38.661533 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:38.661538 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:38.661579 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:38.663804 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:38.664350 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:38.664353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:38.664355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:38.664459 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:38.664740 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:38.664743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:38.664744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:38.665014 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:38.665179 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:38.665422 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 19:03:38.665632 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 19:03:38.669466 15907 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:03:38.676038 15907 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:03:38.705386 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:38.689068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40736 (local address 127.0.97.1:44949)
0924 19:03:38.690219 (+ 1151us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:38.690230 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:38.692620 (+ 2390us) server_negotiation.cc:407] Connection header received
0924 19:03:38.692774 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:38.692786 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:38.693126 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:38.693353 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:03:38.695903 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.698074 (+ 2171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.702595 (+ 4521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.704247 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.704381 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:38.704621 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:38.704904 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 19:03:38.705059 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":953,"thread_start_us":800,"threads_started":1}
I20260924 19:03:38.705464 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:38.688441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44949 (local address 127.0.0.1:40736)
0924 19:03:38.691347 (+ 2906us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:38.691397 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:38.691622 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:38.693754 (+ 2132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:38.693765 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:38.693809 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:03:38.694341 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:38.694357 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.698295 (+ 3938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.698304 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:38.702404 (+ 4100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:38.702423 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.704439 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.704445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:38.704672 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:38.704727 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 19:03:38.704847 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 19:03:38.705034 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2603,"mutex_wait_us":43,"thread_start_us":11798,"threads_started":1}
I20260924 19:03:38.706581 16381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:38.688697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44109 (local address 127.0.0.1:55606)
0924 19:03:38.697867 (+ 9170us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:38.697882 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:38.698022 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:38.699087 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:38.699095 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:38.699130 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:03:38.699703 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:38.699717 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.701532 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.701537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:38.703781 (+ 2244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:38.703799 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.705782 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.705789 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:38.705955 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:38.705992 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 19:03:38.706082 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 19:03:38.706203 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9051,"mutex_wait_us":22,"thread_start_us":8891,"threads_started":1}
I20260924 19:03:38.706590 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:38.688267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55606 (local address 127.0.97.3:44109)
0924 19:03:38.689652 (+ 1385us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:38.689665 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:38.697995 (+ 8330us) server_negotiation.cc:407] Connection header received
0924 19:03:38.698171 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:38.698183 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:38.698485 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:38.698709 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:03:38.699962 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.701390 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.703953 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.705280 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.705403 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:38.705649 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:38.706151 (+ 502us) server_negotiation.cc:299] Negotiation successful
0924 19:03:38.706304 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1163,"thread_start_us":966,"threads_started":1}
I20260924 19:03:38.709431 16033 tablet_service.cc:1467] Processing CreateTablet for tablet be8781bac44b4da98b35b598af25b6b8 (DEFAULT_TABLE table=test-table [id=70ea7a2c1f8042c9b9ca2c851d5899d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:03:38.710021 16300 tablet_service.cc:1467] Processing CreateTablet for tablet be8781bac44b4da98b35b598af25b6b8 (DEFAULT_TABLE table=test-table [id=70ea7a2c1f8042c9b9ca2c851d5899d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:03:38.710768 16033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be8781bac44b4da98b35b598af25b6b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:03:38.711213 16300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be8781bac44b4da98b35b598af25b6b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:03:38.715101 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:38.688150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41921 (local address 127.0.0.1:47400)
0924 19:03:38.700547 (+ 12397us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:38.700565 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:38.700676 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:38.701255 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:38.701260 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:38.701282 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:03:38.701785 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:38.701796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.703475 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.703479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:38.705553 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:38.705566 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.714238 (+ 8672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.714244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:38.714412 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:38.714474 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 19:03:38.714559 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 19:03:38.714688 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11600,"mutex_wait_us":1,"thread_start_us":2029,"threads_started":1}
I20260924 19:03:38.715875 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:38.688518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47400 (local address 127.0.97.2:41921)
0924 19:03:38.700166 (+ 11648us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:38.700175 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:38.700667 (+ 492us) server_negotiation.cc:407] Connection header received
0924 19:03:38.700798 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:38.700806 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:38.701031 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:38.701251 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:03:38.701976 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.703307 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.705721 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.714014 (+ 8293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.714166 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:38.714363 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:38.715405 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0924 19:03:38.715581 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11481,"thread_start_us":11380,"threads_started":1}
I20260924 19:03:38.719589 16167 tablet_service.cc:1467] Processing CreateTablet for tablet be8781bac44b4da98b35b598af25b6b8 (DEFAULT_TABLE table=test-table [id=70ea7a2c1f8042c9b9ca2c851d5899d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:03:38.720958 16167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be8781bac44b4da98b35b598af25b6b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:03:38.723501 16384 tablet_bootstrap.cc:492] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485: Bootstrap starting.
I20260924 19:03:38.726204 16385 tablet_bootstrap.cc:492] T be8781bac44b4da98b35b598af25b6b8 P 4db698cbb4b045b98841a62ac7a61b49: Bootstrap starting.
I20260924 19:03:38.726810 16384 tablet_bootstrap.cc:654] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485: Neither blocks nor log segments found. Creating new log.
I20260924 19:03:38.728098 16384 log.cc:824] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485: Log is configured to *not* fsync() on all Append() calls
I20260924 19:03:38.729560 16385 tablet_bootstrap.cc:654] T be8781bac44b4da98b35b598af25b6b8 P 4db698cbb4b045b98841a62ac7a61b49: Neither blocks nor log segments found. Creating new log.
I20260924 19:03:38.730736 16385 log.cc:824] T be8781bac44b4da98b35b598af25b6b8 P 4db698cbb4b045b98841a62ac7a61b49: Log is configured to *not* fsync() on all Append() calls
I20260924 19:03:38.731302 16384 tablet_bootstrap.cc:492] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485: No bootstrap required, opened a new log
I20260924 19:03:38.731688 16384 ts_tablet_manager.cc:1397] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.002s
I20260924 19:03:38.735107 16388 tablet_bootstrap.cc:492] T be8781bac44b4da98b35b598af25b6b8 P 54662ec87ba34e70a719d89f450b8d1e: Bootstrap starting.
I20260924 19:03:38.736092 16385 tablet_bootstrap.cc:492] T be8781bac44b4da98b35b598af25b6b8 P 4db698cbb4b045b98841a62ac7a61b49: No bootstrap required, opened a new log
I20260924 19:03:38.736341 16385 ts_tablet_manager.cc:1397] T be8781bac44b4da98b35b598af25b6b8 P 4db698cbb4b045b98841a62ac7a61b49: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 19:03:38.738036 16388 tablet_bootstrap.cc:654] T be8781bac44b4da98b35b598af25b6b8 P 54662ec87ba34e70a719d89f450b8d1e: Neither blocks nor log segments found. Creating new log.
I20260924 19:03:38.739178 16388 log.cc:824] T be8781bac44b4da98b35b598af25b6b8 P 54662ec87ba34e70a719d89f450b8d1e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:03:38.741847 16388 tablet_bootstrap.cc:492] T be8781bac44b4da98b35b598af25b6b8 P 54662ec87ba34e70a719d89f450b8d1e: No bootstrap required, opened a new log
I20260924 19:03:38.742153 16388 ts_tablet_manager.cc:1397] T be8781bac44b4da98b35b598af25b6b8 P 54662ec87ba34e70a719d89f450b8d1e: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260924 19:03:38.742450 16384 raft_consensus.cc:348] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db698cbb4b045b98841a62ac7a61b49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44109 } } peers { permanent_uuid: "54662ec87ba34e70a719d89f450b8d1e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41921 } } peers { permanent_uuid: "5c688e4544644d3aa51249bdd95e0485" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44949 } }
I20260924 19:03:38.742879 16384 raft_consensus.cc:374] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:03:38.743013 16384 raft_consensus.cc:729] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c688e4544644d3aa51249bdd95e0485, State: Initialized, Role: FOLLOWER
I20260924 19:03:38.743634 16384 consensus_queue.cc:260] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db698cbb4b045b98841a62ac7a61b49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44109 } } peers { permanent_uuid: "54662ec87ba34e70a719d89f450b8d1e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41921 } } peers { permanent_uuid: "5c688e4544644d3aa51249bdd95e0485" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44949 } }
I20260924 19:03:38.745321 16384 ts_tablet_manager.cc:1428] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20260924 19:03:38.747952 16385 raft_consensus.cc:348] T be8781bac44b4da98b35b598af25b6b8 P 4db698cbb4b045b98841a62ac7a61b49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db698cbb4b045b98841a62ac7a61b49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44109 } } peers { permanent_uuid: "54662ec87ba34e70a719d89f450b8d1e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41921 } } peers { permanent_uuid: "5c688e4544644d3aa51249bdd95e0485" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44949 } }
I20260924 19:03:38.748379 16385 raft_consensus.cc:374] T be8781bac44b4da98b35b598af25b6b8 P 4db698cbb4b045b98841a62ac7a61b49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:03:38.748469 16385 raft_consensus.cc:729] T be8781bac44b4da98b35b598af25b6b8 P 4db698cbb4b045b98841a62ac7a61b49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4db698cbb4b045b98841a62ac7a61b49, State: Initialized, Role: FOLLOWER
I20260924 19:03:38.749050 16385 consensus_queue.cc:260] T be8781bac44b4da98b35b598af25b6b8 P 4db698cbb4b045b98841a62ac7a61b49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db698cbb4b045b98841a62ac7a61b49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44109 } } peers { permanent_uuid: "54662ec87ba34e70a719d89f450b8d1e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41921 } } peers { permanent_uuid: "5c688e4544644d3aa51249bdd95e0485" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44949 } }
I20260924 19:03:38.750079 16365 heartbeater.cc:503] Master 127.0.97.62:42697 was elected leader, sending a full tablet report...
I20260924 19:03:38.750667 16385 ts_tablet_manager.cc:1428] T be8781bac44b4da98b35b598af25b6b8 P 4db698cbb4b045b98841a62ac7a61b49: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260924 19:03:38.751627 16388 raft_consensus.cc:348] T be8781bac44b4da98b35b598af25b6b8 P 54662ec87ba34e70a719d89f450b8d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db698cbb4b045b98841a62ac7a61b49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44109 } } peers { permanent_uuid: "54662ec87ba34e70a719d89f450b8d1e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41921 } } peers { permanent_uuid: "5c688e4544644d3aa51249bdd95e0485" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44949 } }
I20260924 19:03:38.752007 16388 raft_consensus.cc:374] T be8781bac44b4da98b35b598af25b6b8 P 54662ec87ba34e70a719d89f450b8d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:03:38.752120 16388 raft_consensus.cc:729] T be8781bac44b4da98b35b598af25b6b8 P 54662ec87ba34e70a719d89f450b8d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54662ec87ba34e70a719d89f450b8d1e, State: Initialized, Role: FOLLOWER
I20260924 19:03:38.752735 16388 consensus_queue.cc:260] T be8781bac44b4da98b35b598af25b6b8 P 54662ec87ba34e70a719d89f450b8d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db698cbb4b045b98841a62ac7a61b49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44109 } } peers { permanent_uuid: "54662ec87ba34e70a719d89f450b8d1e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41921 } } peers { permanent_uuid: "5c688e4544644d3aa51249bdd95e0485" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44949 } }
I20260924 19:03:38.753780 16232 heartbeater.cc:503] Master 127.0.97.62:42697 was elected leader, sending a full tablet report...
I20260924 19:03:38.754263 16388 ts_tablet_manager.cc:1428] T be8781bac44b4da98b35b598af25b6b8 P 54662ec87ba34e70a719d89f450b8d1e: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260924 19:03:38.795109 16390 raft_consensus.cc:482] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:03:38.795423 16390 raft_consensus.cc:504] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db698cbb4b045b98841a62ac7a61b49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44109 } } peers { permanent_uuid: "54662ec87ba34e70a719d89f450b8d1e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41921 } } peers { permanent_uuid: "5c688e4544644d3aa51249bdd95e0485" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44949 } }
I20260924 19:03:38.797091 16390 leader_election.cc:288] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4db698cbb4b045b98841a62ac7a61b49 (127.0.97.3:44109), 54662ec87ba34e70a719d89f450b8d1e (127.0.97.2:41921)
I20260924 19:03:38.804376 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:38.797618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57999 (local address 127.0.97.3:44109)
0924 19:03:38.797832 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:38.797840 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:38.798803 (+ 963us) server_negotiation.cc:407] Connection header received
0924 19:03:38.798943 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:38.798968 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:38.799045 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:38.799155 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:03:38.799744 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.800446 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.802530 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.803405 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.803557 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:38.803668 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:38.804002 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 19:03:38.804123 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 19:03:38.804373 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:38.797618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36791 (local address 127.0.97.2:41921)
0924 19:03:38.797829 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:38.797834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:38.798661 (+ 827us) server_negotiation.cc:407] Connection header received
0924 19:03:38.798808 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:38.798814 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:38.798897 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:38.798989 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:03:38.799709 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.800431 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.802289 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.803393 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.803525 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:38.803631 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:38.804001 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 19:03:38.804119 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:03:38.804666 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:38.797648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41921 (local address 127.0.97.1:36791)
0924 19:03:38.798538 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:38.798554 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:38.798681 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:38.799012 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:38.799017 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:38.799040 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:03:38.799557 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:38.799567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.800556 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.800561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:38.802133 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:38.802145 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.803629 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.803638 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:38.803843 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:38.803874 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 19:03:38.803977 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 19:03:38.804114 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":19,"spinlock_wait_cycles":52096,"thread_start_us":514,"threads_started":1}
I20260924 19:03:38.804674 16394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:38.797648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44109 (local address 127.0.97.1:57999)
0924 19:03:38.798667 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0924 19:03:38.798690 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:03:38.798823 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:03:38.799179 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:03:38.799185 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:03:38.799213 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:03:38.799558 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:38.799565 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.800627 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.800631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:38.802367 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:03:38.802379 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.803664 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.803671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:03:38.803813 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:38.803864 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 19:03:38.803976 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 19:03:38.804105 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":896,"mutex_wait_us":49,"thread_start_us":472,"threads_started":1}
I20260924 19:03:38.805729 16187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be8781bac44b4da98b35b598af25b6b8" candidate_uuid: "5c688e4544644d3aa51249bdd95e0485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54662ec87ba34e70a719d89f450b8d1e" is_pre_election: true
I20260924 19:03:38.805727 16320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be8781bac44b4da98b35b598af25b6b8" candidate_uuid: "5c688e4544644d3aa51249bdd95e0485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4db698cbb4b045b98841a62ac7a61b49" is_pre_election: true
I20260924 19:03:38.806187 16187 raft_consensus.cc:2454] T be8781bac44b4da98b35b598af25b6b8 P 54662ec87ba34e70a719d89f450b8d1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c688e4544644d3aa51249bdd95e0485 in term 0.
I20260924 19:03:38.806201 16320 raft_consensus.cc:2454] T be8781bac44b4da98b35b598af25b6b8 P 4db698cbb4b045b98841a62ac7a61b49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c688e4544644d3aa51249bdd95e0485 in term 0.
I20260924 19:03:38.806857 15989 leader_election.cc:302] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4db698cbb4b045b98841a62ac7a61b49, 54662ec87ba34e70a719d89f450b8d1e, 5c688e4544644d3aa51249bdd95e0485; no voters:
I20260924 19:03:38.807224 16390 raft_consensus.cc:2781] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:03:38.807369 16390 raft_consensus.cc:482] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:03:38.807483 16390 raft_consensus.cc:3037] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:03:38.809706 16390 raft_consensus.cc:504] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db698cbb4b045b98841a62ac7a61b49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44109 } } peers { permanent_uuid: "54662ec87ba34e70a719d89f450b8d1e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41921 } } peers { permanent_uuid: "5c688e4544644d3aa51249bdd95e0485" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44949 } }
I20260924 19:03:38.810349 16390 leader_election.cc:288] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485 [CANDIDATE]: Term 1 election: Requested vote from peers 4db698cbb4b045b98841a62ac7a61b49 (127.0.97.3:44109), 54662ec87ba34e70a719d89f450b8d1e (127.0.97.2:41921)
I20260924 19:03:38.810706 16320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be8781bac44b4da98b35b598af25b6b8" candidate_uuid: "5c688e4544644d3aa51249bdd95e0485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4db698cbb4b045b98841a62ac7a61b49"
I20260924 19:03:38.810842 16187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be8781bac44b4da98b35b598af25b6b8" candidate_uuid: "5c688e4544644d3aa51249bdd95e0485" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54662ec87ba34e70a719d89f450b8d1e"
I20260924 19:03:38.810914 16320 raft_consensus.cc:3037] T be8781bac44b4da98b35b598af25b6b8 P 4db698cbb4b045b98841a62ac7a61b49 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:03:38.811041 16187 raft_consensus.cc:3037] T be8781bac44b4da98b35b598af25b6b8 P 54662ec87ba34e70a719d89f450b8d1e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:03:38.814158 16320 raft_consensus.cc:2454] T be8781bac44b4da98b35b598af25b6b8 P 4db698cbb4b045b98841a62ac7a61b49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c688e4544644d3aa51249bdd95e0485 in term 1.
I20260924 19:03:38.814158 16187 raft_consensus.cc:2454] T be8781bac44b4da98b35b598af25b6b8 P 54662ec87ba34e70a719d89f450b8d1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c688e4544644d3aa51249bdd95e0485 in term 1.
I20260924 19:03:38.814551 15989 leader_election.cc:302] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54662ec87ba34e70a719d89f450b8d1e, 5c688e4544644d3aa51249bdd95e0485; no voters:
I20260924 19:03:38.814952 16390 raft_consensus.cc:2781] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:03:38.815390 16390 raft_consensus.cc:686] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485 [term 1 LEADER]: Becoming Leader. State: Replica: 5c688e4544644d3aa51249bdd95e0485, State: Running, Role: LEADER
I20260924 19:03:38.815940 16390 consensus_queue.cc:237] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db698cbb4b045b98841a62ac7a61b49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44109 } } peers { permanent_uuid: "54662ec87ba34e70a719d89f450b8d1e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41921 } } peers { permanent_uuid: "5c688e4544644d3aa51249bdd95e0485" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44949 } }
I20260924 19:03:38.820180 15907 catalog_manager.cc:5495] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c688e4544644d3aa51249bdd95e0485 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "5c688e4544644d3aa51249bdd95e0485" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4db698cbb4b045b98841a62ac7a61b49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54662ec87ba34e70a719d89f450b8d1e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c688e4544644d3aa51249bdd95e0485" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44949 } health_report { overall_health: HEALTHY } } }
W20260924 19:03:38.854069 16233 tablet.cc:2369] T be8781bac44b4da98b35b598af25b6b8 P 54662ec87ba34e70a719d89f450b8d1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:03:38.856585 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:38.850874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40738 (local address 127.0.97.1:44949)
0924 19:03:38.851068 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:38.851073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:38.851169 (+ 96us) server_negotiation.cc:407] Connection header received
0924 19:03:38.851334 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:38.851337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:38.851429 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:38.851556 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:03:38.851917 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.852727 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.854351 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.854771 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.854904 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:38.855417 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:03:38.855966 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:03:38.856129 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:38.856235 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 19:03:38.856315 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:03:38.864427 16187 raft_consensus.cc:1261] T be8781bac44b4da98b35b598af25b6b8 P 54662ec87ba34e70a719d89f450b8d1e [term 1 FOLLOWER]: Refusing update from remote peer 5c688e4544644d3aa51249bdd95e0485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:03:38.864446 16320 raft_consensus.cc:1261] T be8781bac44b4da98b35b598af25b6b8 P 4db698cbb4b045b98841a62ac7a61b49 [term 1 FOLLOWER]: Refusing update from remote peer 5c688e4544644d3aa51249bdd95e0485: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:03:38.865371 16395 consensus_queue.cc:1035] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4db698cbb4b045b98841a62ac7a61b49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:03:38.865774 16390 consensus_queue.cc:1035] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54662ec87ba34e70a719d89f450b8d1e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:03:38.870549 16099 tablet.cc:2369] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:03:38.876489 16366 tablet.cc:2369] T be8781bac44b4da98b35b598af25b6b8 P 4db698cbb4b045b98841a62ac7a61b49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:03:38.889748 15907 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:03:38.889961 15907 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:03:38.893194 15907 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f: Sending DeleteTablet for 3 replicas of tablet be8781bac44b4da98b35b598af25b6b8
I20260924 19:03:38.894266 16300 tablet_service.cc:1514] Processing DeleteTablet for tablet be8781bac44b4da98b35b598af25b6b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:03:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55606
I20260924 19:03:38.894294 16167 tablet_service.cc:1514] Processing DeleteTablet for tablet be8781bac44b4da98b35b598af25b6b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:03:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47400
I20260924 19:03:38.894527 16033 tablet_service.cc:1514] Processing DeleteTablet for tablet be8781bac44b4da98b35b598af25b6b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:03:38 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40736
I20260924 19:03:38.895421 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 15970
I20260924 19:03:38.905411 16408 tablet_replica.cc:330] T be8781bac44b4da98b35b598af25b6b8 P 54662ec87ba34e70a719d89f450b8d1e: stopping tablet replica
I20260924 19:03:38.905966 16408 raft_consensus.cc:2229] T be8781bac44b4da98b35b598af25b6b8 P 54662ec87ba34e70a719d89f450b8d1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:03:38.906555 16408 raft_consensus.cc:2258] T be8781bac44b4da98b35b598af25b6b8 P 54662ec87ba34e70a719d89f450b8d1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:03:38.911216 16407 tablet_replica.cc:330] T be8781bac44b4da98b35b598af25b6b8 P 4db698cbb4b045b98841a62ac7a61b49: stopping tablet replica
I20260924 19:03:38.911725 16407 raft_consensus.cc:2229] T be8781bac44b4da98b35b598af25b6b8 P 4db698cbb4b045b98841a62ac7a61b49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:03:38.912302 16407 raft_consensus.cc:2258] T be8781bac44b4da98b35b598af25b6b8 P 4db698cbb4b045b98841a62ac7a61b49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:03:38.911823 16408 ts_tablet_manager.cc:1905] T be8781bac44b4da98b35b598af25b6b8 P 54662ec87ba34e70a719d89f450b8d1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:03:38.913350 16409 tablet_replica.cc:330] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485: stopping tablet replica
I20260924 19:03:38.913779 16409 raft_consensus.cc:2229] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:03:38.914486 16409 raft_consensus.cc:2258] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:03:38.915978 16407 ts_tablet_manager.cc:1905] T be8781bac44b4da98b35b598af25b6b8 P 4db698cbb4b045b98841a62ac7a61b49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:03:38.919983 16408 ts_tablet_manager.cc:1918] T be8781bac44b4da98b35b598af25b6b8 P 54662ec87ba34e70a719d89f450b8d1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:03:38.920192 16408 log.cc:1196] T be8781bac44b4da98b35b598af25b6b8 P 54662ec87ba34e70a719d89f450b8d1e: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/wal/wals/be8781bac44b4da98b35b598af25b6b8
I20260924 19:03:38.920658 16409 ts_tablet_manager.cc:1905] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:03:38.920876 16408 ts_tablet_manager.cc:1939] T be8781bac44b4da98b35b598af25b6b8 P 54662ec87ba34e70a719d89f450b8d1e: Deleting consensus metadata
I20260924 19:03:38.922740 15896 catalog_manager.cc:4841] TS 54662ec87ba34e70a719d89f450b8d1e (127.0.97.2:41921): tablet be8781bac44b4da98b35b598af25b6b8 (table test-table [id=70ea7a2c1f8042c9b9ca2c851d5899d2]) successfully deleted
I20260924 19:03:38.923331 16407 ts_tablet_manager.cc:1918] T be8781bac44b4da98b35b598af25b6b8 P 4db698cbb4b045b98841a62ac7a61b49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:03:38.923496 16407 log.cc:1196] T be8781bac44b4da98b35b598af25b6b8 P 4db698cbb4b045b98841a62ac7a61b49: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/wal/wals/be8781bac44b4da98b35b598af25b6b8
I20260924 19:03:38.923990 16407 ts_tablet_manager.cc:1939] T be8781bac44b4da98b35b598af25b6b8 P 4db698cbb4b045b98841a62ac7a61b49: Deleting consensus metadata
I20260924 19:03:38.925697 15895 catalog_manager.cc:4841] TS 4db698cbb4b045b98841a62ac7a61b49 (127.0.97.3:44109): tablet be8781bac44b4da98b35b598af25b6b8 (table test-table [id=70ea7a2c1f8042c9b9ca2c851d5899d2]) successfully deleted
I20260924 19:03:38.927199 16409 ts_tablet_manager.cc:1918] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:03:38.927387 16409 log.cc:1196] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/wal/wals/be8781bac44b4da98b35b598af25b6b8
I20260924 19:03:38.927824 16409 ts_tablet_manager.cc:1939] T be8781bac44b4da98b35b598af25b6b8 P 5c688e4544644d3aa51249bdd95e0485: Deleting consensus metadata
Sep 24 19:03:38 dist-test-slave-9fn1 krb5kdc[15864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 19:03:38.929334 15893 catalog_manager.cc:4841] TS 5c688e4544644d3aa51249bdd95e0485 (127.0.97.1:44949): tablet be8781bac44b4da98b35b598af25b6b8 (table test-table [id=70ea7a2c1f8042c9b9ca2c851d5899d2]) successfully deleted
I20260924 19:03:38.936249 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:38.896936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40750 (local address 127.0.97.1:44949)
0924 19:03:38.897111 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:38.897118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:38.910211 (+ 13093us) server_negotiation.cc:407] Connection header received
0924 19:03:38.910500 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:38.910505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:38.910597 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:38.910709 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:38.914570 (+ 3861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.915762 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.917718 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:38.918396 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:38.918500 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:38.930184 (+ 11684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:38.930196 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:38.930208 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:38.930243 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:38.933324 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:38.934098 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:38.934108 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:38.934111 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:38.934255 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:38.934663 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:38.934667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:38.934672 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:38.935006 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:38.935326 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:38.935670 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 19:03:38.935937 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 19:03:38.936952 16093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:03:39.034806 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 15970
I20260924 19:03:39.059098 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 16104
Sep 24 19:03:39 dist-test-slave-9fn1 krb5kdc[15864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 19:03:39.074254 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:39.059824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47408 (local address 127.0.97.2:41921)
0924 19:03:39.059996 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:39.060002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:39.060024 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:03:39.060194 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:39.060199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:39.060278 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:39.060382 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:39.061680 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:39.062748 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:39.064222 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:39.064840 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:39.064955 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:39.068981 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:39.068995 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:39.069009 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:39.069044 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:39.071900 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:39.072486 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:39.072493 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:39.072495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:39.072626 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:39.072926 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:39.072929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:39.072931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:39.073295 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:39.073725 (+ 430us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:39.073851 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 19:03:39.074005 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 19:03:39.074875 16227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:03:39.162462 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 16104
I20260924 19:03:39.179422 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 16237
Sep 24 19:03:39 dist-test-slave-9fn1 krb5kdc[15864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 19:03:39.193969 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:03:39.180202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55612 (local address 127.0.97.3:44109)
0924 19:03:39.180369 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 19:03:39.180373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:03:39.180459 (+ 86us) server_negotiation.cc:407] Connection header received
0924 19:03:39.180608 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:03:39.180612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:03:39.180675 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:03:39.180768 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:03:39.182151 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:39.182871 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:39.184134 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:03:39.184534 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:03:39.184631 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:03:39.188944 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:03:39.188959 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:03:39.188975 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:03:39.189006 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:03:39.191754 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:39.192345 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:39.192351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:39.192354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:39.192474 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:03:39.192780 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:03:39.192783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:03:39.192785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:03:39.193102 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:03:39.193309 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 19:03:39.193573 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 19:03:39.193738 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 19:03:39.194600 16360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:03:39.282181 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 16237
I20260924 19:03:39.300246 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 15877
I20260924 19:03:39.301424 15939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:03:39.410912 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 15877
I20260924 19:03:59.432400 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42697
--webserver_interface=127.0.97.62
--webserver_port=36985
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:03:59.759202 16415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:03:59.759482 16415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:03:59.759649 16415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:03:59.768313 16415 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 19:03:59.768419 16415 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 19:03:59.768514 16415 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 19:03:59.768602 16415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:03:59.768679 16415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:03:59.768774 16415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:03:59.768851 16415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:03:59.768930 16415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:03:59.779441 16415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=36985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:03:59.780725 16415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:03:59.782119 16415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:03:59.804194 16422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:59.804309 16424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:03:59.804710 16421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:03:59.805249 16415 server_base.cc:1034] running on GCE node
I20260924 19:03:59.806216 16415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:03:59.807811 16415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:03:59.809065 16415 hybrid_clock.cc:648] HybridClock initialized: now 1790276639808980 us; error 56 us; skew 500 ppm
I20260924 19:03:59.809535 16415 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:03:59 dist-test-slave-9fn1 krb5kdc[15864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:03:59.813182 16415 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:03:59.814894 16415 webserver.cc:460] Webserver started at http://127.0.97.62:36985/ using document root <none> and password file <none>
I20260924 19:03:59.815690 16415 fs_manager.cc:359] Metadata directory not provided
I20260924 19:03:59.815841 16415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:03:59.822301 16415 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 19:03:59.825146 16431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:03:59.826534 16415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 19:03:59.826733 16415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "276693ed963e44a6827ffc8d97d8356f"
format_stamp: "Formatted at 2026-09-24 19:03:36 on dist-test-slave-9fn1"
server_key: "a8c1ce2fe86d58c058774f59c89fa6fd"
server_key_iv: "55309b4e33838f818092f9bca0d7d690"
server_key_version: "encryptionkey@0"
I20260924 19:03:59.827795 16415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:03:59.853736 16415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:03:59.858436 16415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:03:59.858753 16415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:03:59.877718 16415 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42697
I20260924 19:03:59.877737 16482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42697 every 8 connection(s)
I20260924 19:03:59.879299 16415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:03:59.884634 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 16415
I20260924 19:03:59.885672 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:44949
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:42697
--webserver_port=42257
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:03:59.887166 16483 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:03:59.892449 16483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f: Bootstrap starting.
I20260924 19:03:59.913165 16483 log.cc:824] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f: Log is configured to *not* fsync() on all Append() calls
I20260924 19:03:59.924993 16483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:03:59.925652 16483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f: Bootstrap complete.
I20260924 19:03:59.935209 16483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276693ed963e44a6827ffc8d97d8356f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42697 } }
I20260924 19:03:59.936033 16483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 276693ed963e44a6827ffc8d97d8356f, State: Initialized, Role: FOLLOWER
I20260924 19:03:59.936691 16483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276693ed963e44a6827ffc8d97d8356f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42697 } }
I20260924 19:03:59.936928 16483 raft_consensus.cc:388] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:03:59.937045 16483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:03:59.937217 16483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:03:59.940948 16483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276693ed963e44a6827ffc8d97d8356f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42697 } }
I20260924 19:03:59.941486 16483 leader_election.cc:302] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 276693ed963e44a6827ffc8d97d8356f; no voters:
I20260924 19:03:59.942185 16483 leader_election.cc:288] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 19:03:59.942315 16487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:03:59.942992 16487 raft_consensus.cc:686] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [term 2 LEADER]: Becoming Leader. State: Replica: 276693ed963e44a6827ffc8d97d8356f, State: Running, Role: LEADER
I20260924 19:03:59.943540 16487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276693ed963e44a6827ffc8d97d8356f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42697 } }
I20260924 19:03:59.944226 16483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:03:59.946265 16489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "276693ed963e44a6827ffc8d97d8356f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276693ed963e44a6827ffc8d97d8356f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42697 } } }
I20260924 19:03:59.946650 16489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [sys.catalog]: This master's current role is: LEADER
I20260924 19:03:59.946306 16488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 276693ed963e44a6827ffc8d97d8356f. Latest consensus state: current_term: 2 leader_uuid: "276693ed963e44a6827ffc8d97d8356f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276693ed963e44a6827ffc8d97d8356f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42697 } } }
I20260924 19:03:59.946879 16488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [sys.catalog]: This master's current role is: LEADER
I20260924 19:03:59.957626 16492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:03:59.965256 16492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:03:59.966656 16492 catalog_manager.cc:1259] Loaded cluster ID: 71d271b6a53a4395bc084281617a1158
I20260924 19:03:59.966765 16492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:03:59.970788 16492 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:03:59.972420 16492 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f: Loaded TSK: 0
I20260924 19:04:00.015825 16492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f: Generated new TSK 1
I20260924 19:04:00.016251 16492 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 19:04:00.250247 16485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:00.250562 16485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:00.250730 16485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:00.258281 16485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:00.258430 16485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:00.258670 16485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:04:00.268399 16485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:44949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42257
--tserver_master_addrs=127.0.97.62:42697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:00.269728 16485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:00.271090 16485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:00.283907 16510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:00.283895 16511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:00.285184 16513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:00.285746 16485 server_base.cc:1034] running on GCE node
I20260924 19:04:00.286407 16485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:00.287443 16485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:00.288668 16485 hybrid_clock.cc:648] HybridClock initialized: now 1790276640288614 us; error 47 us; skew 500 ppm
I20260924 19:04:00.288982 16485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:04:00 dist-test-slave-9fn1 krb5kdc[15864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:04:00.292304 16485 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 19:04:00.293926 16485 webserver.cc:460] Webserver started at http://127.0.97.1:42257/ using document root <none> and password file <none>
I20260924 19:04:00.294655 16485 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:00.294811 16485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:00.300294 16485 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 19:04:00.303193 16520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:00.304569 16485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:04:00.304730 16485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "5c688e4544644d3aa51249bdd95e0485"
format_stamp: "Formatted at 2026-09-24 19:03:37 on dist-test-slave-9fn1"
server_key: "b7a4105efd9ae35f9d7a5fc0d3e0f65b"
server_key_iv: "2b8b963ddd332d35098011b4d0a70d3c"
server_key_version: "encryptionkey@0"
I20260924 19:04:00.305755 16485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:00.345005 16485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:00.349653 16485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:00.349941 16485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:00.350935 16485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:04:00.352360 16485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:04:00.352483 16485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:00.352609 16485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:04:00.352713 16485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:00.385583 16485 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44949
I20260924 19:04:00.385622 16632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44949 every 8 connection(s)
I20260924 19:04:00.387081 16485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:04:00.392378 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 16485
I20260924 19:04:00.392971 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:41921
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:42697
--webserver_port=43137
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:04:00 dist-test-slave-9fn1 krb5kdc[15864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:04:00.412830 16636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:00.390120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.1:46669)
0924 19:04:00.390988 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:00.391039 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:00.392815 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:00.394464 (+ 1649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:00.394471 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:00.394990 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:04:00.395535 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:00.395552 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:00.397937 (+ 2385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:00.397941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:00.399180 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:00.399192 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:00.400700 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:00.400710 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:00.401034 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:00.401867 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:04:00.401899 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:04:00.404858 (+ 2959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:04:00.408298 (+ 3440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:04:00.408309 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:04:00.408327 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:04:00.408951 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:04:00.409359 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:04:00.409364 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:04:00.409367 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:04:00.409639 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:04:00.410374 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:04:00.410382 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:04:00.410709 (+ 327us) client_negotiation.cc:769] Sending connection context
0924 19:04:00.410964 (+ 255us) client_negotiation.cc:240] Negotiation successful
0924 19:04:00.411524 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":364,"threads_started":1}
I20260924 19:04:00.412835 16635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:00.389953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46669 (local address 127.0.97.62:42697)
0924 19:04:00.390534 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:00.390545 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:00.391178 (+ 633us) server_negotiation.cc:407] Connection header received
0924 19:04:00.393420 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:00.393440 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:00.393950 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:00.394351 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:00.395739 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:00.397776 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:00.399313 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:00.400417 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:00.400636 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:00.405138 (+ 4502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:00.405146 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:00.405161 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:00.405196 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:00.408151 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:00.409068 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:00.409074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:00.409080 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:00.409224 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:00.409746 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:00.409753 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:00.409756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:00.410222 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:00.410423 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:00.411181 (+ 758us) server_negotiation.cc:299] Negotiation successful
0924 19:04:00.411546 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":261,"threads_started":1}
I20260924 19:04:00.416203 16633 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42697
I20260924 19:04:00.416551 16633 heartbeater.cc:464] Registering TS with master...
I20260924 19:04:00.417533 16633 heartbeater.cc:511] Master 127.0.97.62:42697 requested a full tablet report, sending...
I20260924 19:04:00.420131 16448 ts_manager.cc:194] Registered new tserver with Master: 5c688e4544644d3aa51249bdd95e0485 (127.0.97.1:44949)
I20260924 19:04:00.422194 16448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46669
W20260924 19:04:00.727886 16637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:00.728154 16637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:00.728247 16637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:00.735643 16637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:00.735756 16637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:00.735868 16637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:04:00.745524 16637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:41921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=43137
--tserver_master_addrs=127.0.97.62:42697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:00.746809 16637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:00.748030 16637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:00.761214 16642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:00.761144 16645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:00.761190 16643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:00.762377 16637 server_base.cc:1034] running on GCE node
I20260924 19:04:00.762957 16637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:00.763892 16637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:00.765125 16637 hybrid_clock.cc:648] HybridClock initialized: now 1790276640765079 us; error 32 us; skew 500 ppm
I20260924 19:04:00.765542 16637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:04:00 dist-test-slave-9fn1 krb5kdc[15864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:04:00.769654 16637 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 19:04:00.771059 16637 webserver.cc:460] Webserver started at http://127.0.97.2:43137/ using document root <none> and password file <none>
I20260924 19:04:00.771739 16637 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:00.771881 16637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:00.777159 16637 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:04:00.780288 16652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:00.781756 16637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 19:04:00.781944 16637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "54662ec87ba34e70a719d89f450b8d1e"
format_stamp: "Formatted at 2026-09-24 19:03:38 on dist-test-slave-9fn1"
server_key: "c4099ecfadc935f586bd0166104dd6e1"
server_key_iv: "0972755de12e7748cde063fc253c9e6f"
server_key_version: "encryptionkey@0"
I20260924 19:04:00.782938 16637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:00.793485 16637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:00.797237 16637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:00.797578 16637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:00.798504 16637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:04:00.799729 16637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:04:00.799855 16637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:00.799988 16637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:04:00.800083 16637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:00.833719 16637 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41921
I20260924 19:04:00.833799 16764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41921 every 8 connection(s)
I20260924 19:04:00.835392 16637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:04:00.838478 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 16637
I20260924 19:04:00.839270 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44109
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:42697
--webserver_port=43851
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:04:00 dist-test-slave-9fn1 krb5kdc[15864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:04:00.855243 16635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:00.837945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36857 (local address 127.0.97.62:42697)
0924 19:04:00.838143 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:00.838147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:00.839451 (+ 1304us) server_negotiation.cc:407] Connection header received
0924 19:04:00.841303 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:00.841308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:00.841398 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:00.841557 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:00.843023 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:00.843714 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:00.845386 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:00.845885 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:00.846021 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:00.849379 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:00.849383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:00.849387 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:00.849419 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:00.851912 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:00.852702 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:00.852708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:00.852711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:00.852834 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:00.853359 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:00.853364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:00.853367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:00.853805 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:00.853959 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:00.854653 (+ 694us) server_negotiation.cc:299] Negotiation successful
0924 19:04:00.854878 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 19:04:00.856217 16767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:00.838512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.2:36857)
0924 19:04:00.839284 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:00.839320 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:00.841034 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:00.841799 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:00.841807 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:00.842316 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:04:00.842856 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:00.842873 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:00.843874 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:00.843883 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:00.845234 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:00.845252 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:00.846116 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:00.846123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:00.846359 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:00.847034 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:04:00.847056 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:04:00.849155 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:04:00.852124 (+ 2969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:04:00.852142 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:04:00.852161 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:04:00.852572 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:04:00.852944 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:04:00.852949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:04:00.852953 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:04:00.853150 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:04:00.853936 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:04:00.853944 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:04:00.854300 (+ 356us) client_negotiation.cc:769] Sending connection context
0924 19:04:00.854573 (+ 273us) client_negotiation.cc:240] Negotiation successful
0924 19:04:00.855140 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":319,"threads_started":1}
I20260924 19:04:00.857774 16765 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42697
I20260924 19:04:00.858136 16765 heartbeater.cc:464] Registering TS with master...
I20260924 19:04:00.858824 16765 heartbeater.cc:511] Master 127.0.97.62:42697 requested a full tablet report, sending...
I20260924 19:04:00.860210 16448 ts_manager.cc:194] Registered new tserver with Master: 54662ec87ba34e70a719d89f450b8d1e (127.0.97.2:41921)
I20260924 19:04:00.860993 16448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36857
W20260924 19:04:01.162629 16768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:01.162950 16768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:01.163130 16768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:01.170253 16768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:01.170401 16768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:01.170603 16768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:04:01.180179 16768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43851
--tserver_master_addrs=127.0.97.62:42697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:01.181474 16768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:01.182801 16768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:01.196102 16774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:01.196177 16773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:01.196177 16776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:01.197296 16768 server_base.cc:1034] running on GCE node
I20260924 19:04:01.197952 16768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:01.198858 16768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:01.200083 16768 hybrid_clock.cc:648] HybridClock initialized: now 1790276641200050 us; error 45 us; skew 500 ppm
I20260924 19:04:01.200385 16768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:04:01 dist-test-slave-9fn1 krb5kdc[15864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:04:01.203449 16768 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 19:04:01.204903 16768 webserver.cc:460] Webserver started at http://127.0.97.3:43851/ using document root <none> and password file <none>
I20260924 19:04:01.205651 16768 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:01.205806 16768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:01.211054 16768 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260924 19:04:01.213658 16783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:01.214942 16768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 19:04:01.215126 16768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "4db698cbb4b045b98841a62ac7a61b49"
format_stamp: "Formatted at 2026-09-24 19:03:38 on dist-test-slave-9fn1"
server_key: "3e71cc6f3d136d932d619dd1ec3b42bc"
server_key_iv: "1817b158ce5782877eade30f2c36cf4c"
server_key_version: "encryptionkey@0"
I20260924 19:04:01.216073 16768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:01.233714 16768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:01.237988 16768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:01.238262 16768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:01.239203 16768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:04:01.240446 16768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:04:01.240572 16768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:01.240703 16768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:04:01.240790 16768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:01.272977 16768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44109
I20260924 19:04:01.273063 16895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44109 every 8 connection(s)
I20260924 19:04:01.274458 16768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:04:01.281924 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 16768
Sep 24 19:04:01 dist-test-slave-9fn1 krb5kdc[15864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:04:01.294694 16635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:01.276998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48263 (local address 127.0.97.62:42697)
0924 19:04:01.277184 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:01.277189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:01.278287 (+ 1098us) server_negotiation.cc:407] Connection header received
0924 19:04:01.280121 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:01.280127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:01.280222 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:01.280332 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:01.281844 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.282816 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.284642 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.285083 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.285217 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:01.289092 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:01.289096 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:01.289099 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:01.289128 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:01.291738 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:01.292427 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:01.292434 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:01.292445 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:01.292565 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:01.292994 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:01.293001 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:01.293008 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:01.293275 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:01.293447 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:01.294130 (+ 683us) server_negotiation.cc:299] Negotiation successful
0924 19:04:01.294329 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:04:01.295990 16898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:01.277384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.3:48263)
0924 19:04:01.278128 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:01.278170 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:01.279853 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:01.280569 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:01.280577 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:01.281071 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:04:01.281678 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:01.281696 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.283018 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.283023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:01.284473 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:01.284485 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.285303 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.285311 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:01.285604 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:01.286290 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:04:01.286311 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:04:01.288884 (+ 2573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:04:01.291877 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:04:01.291885 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:04:01.291898 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:04:01.292301 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:04:01.292687 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:04:01.292692 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:04:01.292696 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:04:01.292874 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:04:01.293410 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:04:01.293418 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:04:01.293773 (+ 355us) client_negotiation.cc:769] Sending connection context
0924 19:04:01.294034 (+ 261us) client_negotiation.cc:240] Negotiation successful
0924 19:04:01.294526 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":341,"threads_started":1}
I20260924 19:04:01.296788 16900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:01.282938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56164 (local address 127.0.97.62:42697)
0924 19:04:01.283728 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:01.283733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:01.283749 (+ 16us) server_negotiation.cc:407] Connection header received
0924 19:04:01.283793 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:01.283796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:01.283853 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:01.283960 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:01.285490 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.286422 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.287786 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.288272 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.288362 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:01.291312 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:01.291316 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:01.291319 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:01.291355 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:01.294020 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:01.294672 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:01.294683 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:01.294686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:01.294849 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:01.295307 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:01.295310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:01.295312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:01.295538 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:01.295713 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:01.296128 (+ 415us) server_negotiation.cc:299] Negotiation successful
0924 19:04:01.296363 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":499,"threads_started":1}
I20260924 19:04:01.297559 16896 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42697
I20260924 19:04:01.297850 16896 heartbeater.cc:464] Registering TS with master...
I20260924 19:04:01.298604 16896 heartbeater.cc:511] Master 127.0.97.62:42697 requested a full tablet report, sending...
I20260924 19:04:01.299939 16448 ts_manager.cc:194] Registered new tserver with Master: 4db698cbb4b045b98841a62ac7a61b49 (127.0.97.3:44109)
I20260924 19:04:01.300707 16448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48263
I20260924 19:04:01.311128 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:04:01.317700 16900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:01.312123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56178 (local address 127.0.97.62:42697)
0924 19:04:01.312353 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:01.312360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:01.312659 (+ 299us) server_negotiation.cc:407] Connection header received
0924 19:04:01.312890 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:01.312893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:01.313027 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:01.313185 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:04:01.313642 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.314339 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.315778 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.316185 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.316319 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:01.316834 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:04:01.317035 (+ 201us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 19:04:01.317386 (+ 351us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56178: token expired
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 19:04:01.317826 16900 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56178: token expired
W20260924 19:04:01.318291 16370 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:42697)
I20260924 19:04:01.331212 16900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:01.318741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56186 (local address 127.0.97.62:42697)
0924 19:04:01.318897 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:01.318904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:01.318920 (+ 16us) server_negotiation.cc:407] Connection header received
0924 19:04:01.319091 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:01.319096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:01.319160 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:01.319257 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:01.320657 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.321353 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.322738 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.323135 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.323269 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:01.325877 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:01.325883 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:01.325890 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:01.325927 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:01.328495 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:01.329163 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:01.329170 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:01.329173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:01.329301 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:01.329808 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:01.329817 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:01.329822 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:01.330192 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:01.330377 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:01.330656 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 19:04:01.330822 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 19:04:01.334506 16448 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:04:01.341879 16448 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:04:01.376395 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:01.353469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44109 (local address 127.0.0.1:52294)
0924 19:04:01.367037 (+ 13568us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:01.367067 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:01.367197 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:01.368444 (+ 1247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:01.368452 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:01.368482 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:01.369126 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:01.369165 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.371210 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.371218 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:01.373454 (+ 2236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:01.373469 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.375403 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.375410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:01.375623 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:01.375676 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 19:04:01.375803 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 19:04:01.375995 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13430,"mutex_wait_us":3,"thread_start_us":13223,"threads_started":1}
I20260924 19:04:01.376726 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:01.353644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52646 (local address 127.0.97.2:41921)
0924 19:04:01.369208 (+ 15564us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:01.369218 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:01.369253 (+ 35us) server_negotiation.cc:407] Connection header received
0924 19:04:01.369354 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:01.369365 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:01.369668 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:01.370445 (+ 777us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:01.370630 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.372015 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.374489 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.375657 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.375794 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:01.375974 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:01.376265 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 19:04:01.376450 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15397,"thread_start_us":15293,"threads_started":1}
I20260924 19:04:01.376780 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:01.353283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41921 (local address 127.0.0.1:52646)
0924 19:04:01.365064 (+ 11781us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:01.365127 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:01.365347 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:01.369865 (+ 4518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:01.369869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:01.369892 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:01.370273 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:01.370288 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.372165 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.372171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:01.374310 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:01.374325 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.375894 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.375901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:01.376066 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:01.376101 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 19:04:01.376211 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 19:04:01.376343 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11424,"thread_start_us":12221,"threads_started":1}
I20260924 19:04:01.377322 16910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:01.354684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52294 (local address 127.0.97.3:44109)
0924 19:04:01.367928 (+ 13244us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:01.367934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:01.367962 (+ 28us) server_negotiation.cc:407] Connection header received
0924 19:04:01.368041 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:01.368046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:01.368235 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:01.368440 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:01.369388 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.371079 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.373684 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.375168 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.375310 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:01.375549 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:01.376778 (+ 1229us) server_negotiation.cc:299] Negotiation successful
0924 19:04:01.376944 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13050,"thread_start_us":12920,"threads_started":1}
I20260924 19:04:01.378288 16911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:01.355232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40970 (local address 127.0.97.1:44949)
0924 19:04:01.367459 (+ 12227us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:01.367466 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:01.369834 (+ 2368us) server_negotiation.cc:407] Connection header received
0924 19:04:01.369928 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:01.369937 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:01.370162 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:01.370350 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:01.371019 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.372393 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.374711 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.376137 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.376250 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:01.376435 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:01.377790 (+ 1355us) server_negotiation.cc:299] Negotiation successful
0924 19:04:01.377958 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12095,"thread_start_us":11964,"threads_started":1}
I20260924 19:04:01.378932 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:01.353911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44949 (local address 127.0.0.1:40970)
0924 19:04:01.369567 (+ 15656us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:01.369591 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:01.369702 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:01.370495 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:01.370500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:01.370529 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:01.370828 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:01.370837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.372542 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.372549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:01.374541 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:01.374558 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.376700 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.376706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:01.377194 (+ 488us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:01.377226 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 19:04:01.377315 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 19:04:01.378630 (+ 1315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15530,"thread_start_us":15056,"threads_started":1}
I20260924 19:04:01.380774 16831 tablet_service.cc:1467] Processing CreateTablet for tablet f9cc6a22292a45c696168e738b56a22d (DEFAULT_TABLE table=test-table [id=a62412a008194c23b21cc1e55c408a60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:04:01.380802 16700 tablet_service.cc:1467] Processing CreateTablet for tablet f9cc6a22292a45c696168e738b56a22d (DEFAULT_TABLE table=test-table [id=a62412a008194c23b21cc1e55c408a60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:04:01.382083 16831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9cc6a22292a45c696168e738b56a22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:04:01.382108 16700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9cc6a22292a45c696168e738b56a22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:04:01.382339 16568 tablet_service.cc:1467] Processing CreateTablet for tablet f9cc6a22292a45c696168e738b56a22d (DEFAULT_TABLE table=test-table [id=a62412a008194c23b21cc1e55c408a60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:04:01.383534 16568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9cc6a22292a45c696168e738b56a22d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:04:01.402033 16912 tablet_bootstrap.cc:492] T f9cc6a22292a45c696168e738b56a22d P 4db698cbb4b045b98841a62ac7a61b49: Bootstrap starting.
I20260924 19:04:01.403038 16913 tablet_bootstrap.cc:492] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e: Bootstrap starting.
I20260924 19:04:01.405539 16912 tablet_bootstrap.cc:654] T f9cc6a22292a45c696168e738b56a22d P 4db698cbb4b045b98841a62ac7a61b49: Neither blocks nor log segments found. Creating new log.
I20260924 19:04:01.406633 16914 tablet_bootstrap.cc:492] T f9cc6a22292a45c696168e738b56a22d P 5c688e4544644d3aa51249bdd95e0485: Bootstrap starting.
I20260924 19:04:01.407076 16913 tablet_bootstrap.cc:654] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e: Neither blocks nor log segments found. Creating new log.
I20260924 19:04:01.407065 16912 log.cc:824] T f9cc6a22292a45c696168e738b56a22d P 4db698cbb4b045b98841a62ac7a61b49: Log is configured to *not* fsync() on all Append() calls
I20260924 19:04:01.408311 16913 log.cc:824] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:04:01.410415 16914 tablet_bootstrap.cc:654] T f9cc6a22292a45c696168e738b56a22d P 5c688e4544644d3aa51249bdd95e0485: Neither blocks nor log segments found. Creating new log.
I20260924 19:04:01.411262 16913 tablet_bootstrap.cc:492] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e: No bootstrap required, opened a new log
I20260924 19:04:01.411264 16912 tablet_bootstrap.cc:492] T f9cc6a22292a45c696168e738b56a22d P 4db698cbb4b045b98841a62ac7a61b49: No bootstrap required, opened a new log
I20260924 19:04:01.411372 16914 log.cc:824] T f9cc6a22292a45c696168e738b56a22d P 5c688e4544644d3aa51249bdd95e0485: Log is configured to *not* fsync() on all Append() calls
I20260924 19:04:01.411537 16913 ts_tablet_manager.cc:1397] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 19:04:01.411535 16912 ts_tablet_manager.cc:1397] T f9cc6a22292a45c696168e738b56a22d P 4db698cbb4b045b98841a62ac7a61b49: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260924 19:04:01.414055 16914 tablet_bootstrap.cc:492] T f9cc6a22292a45c696168e738b56a22d P 5c688e4544644d3aa51249bdd95e0485: No bootstrap required, opened a new log
I20260924 19:04:01.414330 16914 ts_tablet_manager.cc:1397] T f9cc6a22292a45c696168e738b56a22d P 5c688e4544644d3aa51249bdd95e0485: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 19:04:01.421725 16914 raft_consensus.cc:348] T f9cc6a22292a45c696168e738b56a22d P 5c688e4544644d3aa51249bdd95e0485 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54662ec87ba34e70a719d89f450b8d1e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41921 } } peers { permanent_uuid: "4db698cbb4b045b98841a62ac7a61b49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44109 } } peers { permanent_uuid: "5c688e4544644d3aa51249bdd95e0485" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44949 } }
I20260924 19:04:01.422132 16914 raft_consensus.cc:374] T f9cc6a22292a45c696168e738b56a22d P 5c688e4544644d3aa51249bdd95e0485 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:04:01.422240 16914 raft_consensus.cc:729] T f9cc6a22292a45c696168e738b56a22d P 5c688e4544644d3aa51249bdd95e0485 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c688e4544644d3aa51249bdd95e0485, State: Initialized, Role: FOLLOWER
I20260924 19:04:01.422394 16912 raft_consensus.cc:348] T f9cc6a22292a45c696168e738b56a22d P 4db698cbb4b045b98841a62ac7a61b49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54662ec87ba34e70a719d89f450b8d1e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41921 } } peers { permanent_uuid: "4db698cbb4b045b98841a62ac7a61b49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44109 } } peers { permanent_uuid: "5c688e4544644d3aa51249bdd95e0485" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44949 } }
I20260924 19:04:01.422392 16913 raft_consensus.cc:348] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54662ec87ba34e70a719d89f450b8d1e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41921 } } peers { permanent_uuid: "4db698cbb4b045b98841a62ac7a61b49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44109 } } peers { permanent_uuid: "5c688e4544644d3aa51249bdd95e0485" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44949 } }
I20260924 19:04:01.422825 16913 raft_consensus.cc:374] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:04:01.422825 16912 raft_consensus.cc:374] T f9cc6a22292a45c696168e738b56a22d P 4db698cbb4b045b98841a62ac7a61b49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:04:01.422925 16913 raft_consensus.cc:729] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54662ec87ba34e70a719d89f450b8d1e, State: Initialized, Role: FOLLOWER
I20260924 19:04:01.422925 16912 raft_consensus.cc:729] T f9cc6a22292a45c696168e738b56a22d P 4db698cbb4b045b98841a62ac7a61b49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4db698cbb4b045b98841a62ac7a61b49, State: Initialized, Role: FOLLOWER
I20260924 19:04:01.422950 16914 consensus_queue.cc:260] T f9cc6a22292a45c696168e738b56a22d P 5c688e4544644d3aa51249bdd95e0485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54662ec87ba34e70a719d89f450b8d1e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41921 } } peers { permanent_uuid: "4db698cbb4b045b98841a62ac7a61b49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44109 } } peers { permanent_uuid: "5c688e4544644d3aa51249bdd95e0485" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44949 } }
I20260924 19:04:01.423497 16912 consensus_queue.cc:260] T f9cc6a22292a45c696168e738b56a22d P 4db698cbb4b045b98841a62ac7a61b49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54662ec87ba34e70a719d89f450b8d1e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41921 } } peers { permanent_uuid: "4db698cbb4b045b98841a62ac7a61b49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44109 } } peers { permanent_uuid: "5c688e4544644d3aa51249bdd95e0485" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44949 } }
I20260924 19:04:01.423506 16913 consensus_queue.cc:260] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54662ec87ba34e70a719d89f450b8d1e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41921 } } peers { permanent_uuid: "4db698cbb4b045b98841a62ac7a61b49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44109 } } peers { permanent_uuid: "5c688e4544644d3aa51249bdd95e0485" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44949 } }
I20260924 19:04:01.424346 16633 heartbeater.cc:503] Master 127.0.97.62:42697 was elected leader, sending a full tablet report...
I20260924 19:04:01.425016 16914 ts_tablet_manager.cc:1428] T f9cc6a22292a45c696168e738b56a22d P 5c688e4544644d3aa51249bdd95e0485: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260924 19:04:01.425364 16913 ts_tablet_manager.cc:1428] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260924 19:04:01.425920 16765 heartbeater.cc:503] Master 127.0.97.62:42697 was elected leader, sending a full tablet report...
I20260924 19:04:01.426261 16896 heartbeater.cc:503] Master 127.0.97.62:42697 was elected leader, sending a full tablet report...
I20260924 19:04:01.426785 16912 ts_tablet_manager.cc:1428] T f9cc6a22292a45c696168e738b56a22d P 4db698cbb4b045b98841a62ac7a61b49: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260924 19:04:01.526070 16897 tablet.cc:2369] T f9cc6a22292a45c696168e738b56a22d P 4db698cbb4b045b98841a62ac7a61b49: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:04:01.587368 16766 tablet.cc:2369] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:04:01.639511 16634 tablet.cc:2369] T f9cc6a22292a45c696168e738b56a22d P 5c688e4544644d3aa51249bdd95e0485: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:04:01.642354 16919 raft_consensus.cc:482] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:04:01.642673 16919 raft_consensus.cc:504] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54662ec87ba34e70a719d89f450b8d1e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41921 } } peers { permanent_uuid: "4db698cbb4b045b98841a62ac7a61b49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44109 } } peers { permanent_uuid: "5c688e4544644d3aa51249bdd95e0485" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44949 } }
I20260924 19:04:01.644388 16919 leader_election.cc:288] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4db698cbb4b045b98841a62ac7a61b49 (127.0.97.3:44109), 5c688e4544644d3aa51249bdd95e0485 (127.0.97.1:44949)
I20260924 19:04:01.652932 16910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:01.644883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56415 (local address 127.0.97.3:44109)
0924 19:04:01.645083 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:01.645089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:01.645946 (+ 857us) server_negotiation.cc:407] Connection header received
0924 19:04:01.646101 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:01.646105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:01.646165 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:01.646245 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:01.646782 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.647840 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.650693 (+ 2853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.651965 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.652105 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:01.652206 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:01.652510 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 19:04:01.652673 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 19:04:01.653090 16922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:01.644781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44109 (local address 127.0.97.2:56415)
0924 19:04:01.645765 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:01.645789 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:01.645985 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:01.646279 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:01.646283 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:01.646303 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:01.646629 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:01.646639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.647972 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.647977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:01.650521 (+ 2544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:01.650537 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.652222 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.652226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:01.652345 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:01.652370 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 19:04:01.652450 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 19:04:01.652601 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":826,"mutex_wait_us":121,"spinlock_wait_cycles":23936,"thread_start_us":520,"threads_started":1}
I20260924 19:04:01.653268 16921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:01.644820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44949 (local address 127.0.97.2:49107)
0924 19:04:01.645788 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:01.645802 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:01.645954 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:01.646616 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:01.646620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:01.646640 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:01.646968 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:01.646987 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.648232 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.648237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:01.650521 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:01.650537 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.652357 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.652362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:01.652569 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:01.652611 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 19:04:01.652702 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 19:04:01.652829 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"mutex_wait_us":80,"thread_start_us":476,"threads_started":1}
I20260924 19:04:01.653602 16911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:01.644883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49107 (local address 127.0.97.1:44949)
0924 19:04:01.645083 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:01.645089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:01.645896 (+ 807us) server_negotiation.cc:407] Connection header received
0924 19:04:01.646408 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:01.646412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:01.646483 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:01.646549 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:01.647134 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.648109 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.650693 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.651976 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.652115 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:01.652205 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:01.652741 (+ 536us) server_negotiation.cc:299] Negotiation successful
0924 19:04:01.652885 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 19:04:01.654261 16851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9cc6a22292a45c696168e738b56a22d" candidate_uuid: "54662ec87ba34e70a719d89f450b8d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4db698cbb4b045b98841a62ac7a61b49" is_pre_election: true
I20260924 19:04:01.654512 16588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9cc6a22292a45c696168e738b56a22d" candidate_uuid: "54662ec87ba34e70a719d89f450b8d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c688e4544644d3aa51249bdd95e0485" is_pre_election: true
I20260924 19:04:01.654769 16851 raft_consensus.cc:2454] T f9cc6a22292a45c696168e738b56a22d P 4db698cbb4b045b98841a62ac7a61b49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54662ec87ba34e70a719d89f450b8d1e in term 0.
I20260924 19:04:01.655079 16588 raft_consensus.cc:2454] T f9cc6a22292a45c696168e738b56a22d P 5c688e4544644d3aa51249bdd95e0485 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54662ec87ba34e70a719d89f450b8d1e in term 0.
I20260924 19:04:01.655520 16655 leader_election.cc:302] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4db698cbb4b045b98841a62ac7a61b49, 54662ec87ba34e70a719d89f450b8d1e; no voters:
I20260924 19:04:01.655956 16919 raft_consensus.cc:2781] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:04:01.656098 16919 raft_consensus.cc:482] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:04:01.656217 16919 raft_consensus.cc:3037] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:04:01.660099 16919 raft_consensus.cc:504] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54662ec87ba34e70a719d89f450b8d1e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41921 } } peers { permanent_uuid: "4db698cbb4b045b98841a62ac7a61b49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44109 } } peers { permanent_uuid: "5c688e4544644d3aa51249bdd95e0485" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44949 } }
I20260924 19:04:01.660903 16919 leader_election.cc:288] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e [CANDIDATE]: Term 1 election: Requested vote from peers 4db698cbb4b045b98841a62ac7a61b49 (127.0.97.3:44109), 5c688e4544644d3aa51249bdd95e0485 (127.0.97.1:44949)
I20260924 19:04:01.661337 16851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9cc6a22292a45c696168e738b56a22d" candidate_uuid: "54662ec87ba34e70a719d89f450b8d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4db698cbb4b045b98841a62ac7a61b49"
I20260924 19:04:01.661346 16588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9cc6a22292a45c696168e738b56a22d" candidate_uuid: "54662ec87ba34e70a719d89f450b8d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c688e4544644d3aa51249bdd95e0485"
I20260924 19:04:01.661600 16851 raft_consensus.cc:3037] T f9cc6a22292a45c696168e738b56a22d P 4db698cbb4b045b98841a62ac7a61b49 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:04:01.661600 16588 raft_consensus.cc:3037] T f9cc6a22292a45c696168e738b56a22d P 5c688e4544644d3aa51249bdd95e0485 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:04:01.665059 16588 raft_consensus.cc:2454] T f9cc6a22292a45c696168e738b56a22d P 5c688e4544644d3aa51249bdd95e0485 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54662ec87ba34e70a719d89f450b8d1e in term 1.
I20260924 19:04:01.665059 16851 raft_consensus.cc:2454] T f9cc6a22292a45c696168e738b56a22d P 4db698cbb4b045b98841a62ac7a61b49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54662ec87ba34e70a719d89f450b8d1e in term 1.
I20260924 19:04:01.665572 16653 leader_election.cc:302] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54662ec87ba34e70a719d89f450b8d1e, 5c688e4544644d3aa51249bdd95e0485; no voters:
I20260924 19:04:01.665931 16919 raft_consensus.cc:2781] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:04:01.666424 16919 raft_consensus.cc:686] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e [term 1 LEADER]: Becoming Leader. State: Replica: 54662ec87ba34e70a719d89f450b8d1e, State: Running, Role: LEADER
I20260924 19:04:01.666911 16919 consensus_queue.cc:237] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54662ec87ba34e70a719d89f450b8d1e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41921 } } peers { permanent_uuid: "4db698cbb4b045b98841a62ac7a61b49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44109 } } peers { permanent_uuid: "5c688e4544644d3aa51249bdd95e0485" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44949 } }
I20260924 19:04:01.671706 16448 catalog_manager.cc:5495] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 54662ec87ba34e70a719d89f450b8d1e (127.0.97.2). New cstate: current_term: 1 leader_uuid: "54662ec87ba34e70a719d89f450b8d1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54662ec87ba34e70a719d89f450b8d1e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4db698cbb4b045b98841a62ac7a61b49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c688e4544644d3aa51249bdd95e0485" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44949 } health_report { overall_health: UNKNOWN } } }
I20260924 19:04:01.753053 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:01.747488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52648 (local address 127.0.97.2:41921)
0924 19:04:01.747749 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:01.747753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:01.747776 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:04:01.748006 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:01.748010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:01.748087 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:01.748212 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:04:01.748539 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.749267 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.750685 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.751080 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.751173 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:01.751641 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:04:01.752340 (+ 699us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:04:01.752470 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:01.752602 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 19:04:01.752699 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260924 19:04:01.762297 16588 raft_consensus.cc:1261] T f9cc6a22292a45c696168e738b56a22d P 5c688e4544644d3aa51249bdd95e0485 [term 1 FOLLOWER]: Refusing update from remote peer 54662ec87ba34e70a719d89f450b8d1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:04:01.762297 16851 raft_consensus.cc:1261] T f9cc6a22292a45c696168e738b56a22d P 4db698cbb4b045b98841a62ac7a61b49 [term 1 FOLLOWER]: Refusing update from remote peer 54662ec87ba34e70a719d89f450b8d1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:04:01.763490 16919 consensus_queue.cc:1035] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4db698cbb4b045b98841a62ac7a61b49" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:04:01.763867 16923 consensus_queue.cc:1035] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c688e4544644d3aa51249bdd95e0485" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:04:01.799274 16448 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:04:01.799540 16448 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:04:01.802784 16448 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f: Sending DeleteTablet for 3 replicas of tablet f9cc6a22292a45c696168e738b56a22d
I20260924 19:04:01.803960 16700 tablet_service.cc:1514] Processing DeleteTablet for tablet f9cc6a22292a45c696168e738b56a22d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:04:01 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52646
I20260924 19:04:01.803953 16831 tablet_service.cc:1514] Processing DeleteTablet for tablet f9cc6a22292a45c696168e738b56a22d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:04:01 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52294
I20260924 19:04:01.804293 16568 tablet_service.cc:1514] Processing DeleteTablet for tablet f9cc6a22292a45c696168e738b56a22d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:04:01 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:40970
I20260924 19:04:01.805457 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 16485
I20260924 19:04:01.818156 16936 tablet_replica.cc:330] T f9cc6a22292a45c696168e738b56a22d P 4db698cbb4b045b98841a62ac7a61b49: stopping tablet replica
I20260924 19:04:01.818801 16937 tablet_replica.cc:330] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e: stopping tablet replica
I20260924 19:04:01.819243 16937 raft_consensus.cc:2229] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:04:01.819799 16938 tablet_replica.cc:330] T f9cc6a22292a45c696168e738b56a22d P 5c688e4544644d3aa51249bdd95e0485: stopping tablet replica
I20260924 19:04:01.820040 16937 raft_consensus.cc:2258] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:04:01.820320 16938 raft_consensus.cc:2229] T f9cc6a22292a45c696168e738b56a22d P 5c688e4544644d3aa51249bdd95e0485 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:04:01.820788 16936 raft_consensus.cc:2229] T f9cc6a22292a45c696168e738b56a22d P 4db698cbb4b045b98841a62ac7a61b49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:04:01.820935 16938 raft_consensus.cc:2258] T f9cc6a22292a45c696168e738b56a22d P 5c688e4544644d3aa51249bdd95e0485 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:04:01.821233 16936 raft_consensus.cc:2258] T f9cc6a22292a45c696168e738b56a22d P 4db698cbb4b045b98841a62ac7a61b49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:04:01.823591 16936 ts_tablet_manager.cc:1905] T f9cc6a22292a45c696168e738b56a22d P 4db698cbb4b045b98841a62ac7a61b49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:04:01.823602 16937 ts_tablet_manager.cc:1905] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:04:01.830659 16936 ts_tablet_manager.cc:1918] T f9cc6a22292a45c696168e738b56a22d P 4db698cbb4b045b98841a62ac7a61b49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:04:01.830921 16936 log.cc:1196] T f9cc6a22292a45c696168e738b56a22d P 4db698cbb4b045b98841a62ac7a61b49: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/wal/wals/f9cc6a22292a45c696168e738b56a22d
I20260924 19:04:01.831660 16936 ts_tablet_manager.cc:1939] T f9cc6a22292a45c696168e738b56a22d P 4db698cbb4b045b98841a62ac7a61b49: Deleting consensus metadata
I20260924 19:04:01.832690 16938 ts_tablet_manager.cc:1905] T f9cc6a22292a45c696168e738b56a22d P 5c688e4544644d3aa51249bdd95e0485: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:04:01.833976 16434 catalog_manager.cc:4841] TS 4db698cbb4b045b98841a62ac7a61b49 (127.0.97.3:44109): tablet f9cc6a22292a45c696168e738b56a22d (table test-table [id=a62412a008194c23b21cc1e55c408a60]) successfully deleted
I20260924 19:04:01.837841 16937 ts_tablet_manager.cc:1918] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:04:01.838094 16937 log.cc:1196] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/wal/wals/f9cc6a22292a45c696168e738b56a22d
I20260924 19:04:01.838752 16937 ts_tablet_manager.cc:1939] T f9cc6a22292a45c696168e738b56a22d P 54662ec87ba34e70a719d89f450b8d1e: Deleting consensus metadata
I20260924 19:04:01.840005 16938 ts_tablet_manager.cc:1918] T f9cc6a22292a45c696168e738b56a22d P 5c688e4544644d3aa51249bdd95e0485: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:04:01.840216 16938 log.cc:1196] T f9cc6a22292a45c696168e738b56a22d P 5c688e4544644d3aa51249bdd95e0485: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/wal/wals/f9cc6a22292a45c696168e738b56a22d
I20260924 19:04:01.840559 16911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:01.806824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40978 (local address 127.0.97.1:44949)
0924 19:04:01.807073 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:01.807082 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:01.822530 (+ 15448us) server_negotiation.cc:407] Connection header received
0924 19:04:01.822743 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:01.822749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:01.822834 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:01.822966 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:01.825084 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.826222 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.828129 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.828750 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.828857 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:01.832707 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:01.832724 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:01.832741 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:01.832780 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:01.836211 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:01.836960 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:01.836967 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:01.836970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:01.837116 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:01.837547 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:01.837555 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:01.837565 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:01.838130 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:01.838581 (+ 451us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:01.839760 (+ 1179us) server_negotiation.cc:299] Negotiation successful
0924 19:04:01.840019 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 19:04:01.840550 16435 catalog_manager.cc:4841] TS 54662ec87ba34e70a719d89f450b8d1e (127.0.97.2:41921): tablet f9cc6a22292a45c696168e738b56a22d (table test-table [id=a62412a008194c23b21cc1e55c408a60]) successfully deleted
I20260924 19:04:01.841177 16938 ts_tablet_manager.cc:1939] T f9cc6a22292a45c696168e738b56a22d P 5c688e4544644d3aa51249bdd95e0485: Deleting consensus metadata
I20260924 19:04:01.841248 16628 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:04:01.963091 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 16485
W20260924 19:04:01.980530 16432 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44949 (error 108)
I20260924 19:04:01.980659 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 16637
I20260924 19:04:01.982862 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:01.982056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44949 (local address 127.0.0.1:40980)
0924 19:04:01.982514 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:01.982645 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":233,"threads_started":1}
W20260924 19:04:01.983053 16432 catalog_manager.cc:4584] TS 5c688e4544644d3aa51249bdd95e0485 (127.0.97.1:44949): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f9cc6a22292a45c696168e738b56a22d: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44949: connect: Connection refused (error 111)
I20260924 19:04:01.996977 16908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:01.981487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52660 (local address 127.0.97.2:41921)
0924 19:04:01.981720 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:01.981727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:01.981826 (+ 99us) server_negotiation.cc:407] Connection header received
0924 19:04:01.982032 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:01.982042 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:01.982129 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:01.982258 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:01.983919 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.985138 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.986713 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:01.987463 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:01.987587 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:01.990217 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:01.990233 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:01.990252 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:01.990289 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:01.994183 (+ 3894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:01.994875 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:01.994884 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:01.994888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:01.995051 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:01.995404 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:01.995409 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:01.995412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:01.995833 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:01.996178 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:01.996335 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 19:04:01.996558 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 19:04:01.997758 16760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:04:02.036873 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:02.036341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44949 (local address 127.0.0.1:40994)
0924 19:04:02.036580 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:02.036702 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260924 19:04:02.037981 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:02.037569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44949 (local address 127.0.0.1:40998)
0924 19:04:02.037737 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:02.037820 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 19:04:02.098523 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:02.097971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44949 (local address 127.0.0.1:41002)
0924 19:04:02.098211 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:02.098324 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260924 19:04:02.099638 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:02.099208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44949 (local address 127.0.0.1:41018)
0924 19:04:02.099361 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:02.099480 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 19:04:02.109838 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 16637
I20260924 19:04:02.129256 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 16768
I20260924 19:04:02.141896 16910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:02.130031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52298 (local address 127.0.97.3:44109)
0924 19:04:02.130197 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:02.130201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:02.130238 (+ 37us) server_negotiation.cc:407] Connection header received
0924 19:04:02.130403 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:02.130407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:02.130473 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:02.130556 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:02.132012 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:02.132748 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:02.134117 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:02.134477 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:02.134567 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:02.137069 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:02.137081 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:02.137097 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:02.137142 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:02.139628 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:02.140209 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:02.140216 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:02.140218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:02.140343 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:02.140681 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:02.140684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:02.140685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:02.141025 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:02.141224 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:02.141447 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 19:04:02.141655 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 19:04:02.142515 16891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:04:02.180706 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:02.180219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44949 (local address 127.0.0.1:41028)
0924 19:04:02.180451 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:02.180542 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 19:04:02.181849 16942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:02.181399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44949 (local address 127.0.0.1:41044)
0924 19:04:02.181644 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:02.181712 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260924 19:04:02.230679 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 16768
I20260924 19:04:02.247048 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 16415
I20260924 19:04:02.248208 16478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:04:02.361282 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 16415
I20260924 19:04:22.385357 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42697
--webserver_interface=127.0.97.62
--webserver_port=36985
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:04:22.715430 16946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:22.715709 16946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:22.715860 16946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:22.723044 16946 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 19:04:22.723140 16946 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 19:04:22.723228 16946 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 19:04:22.723330 16946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:04:22.723428 16946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:22.723546 16946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:22.723636 16946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:04:22.723716 16946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:04:22.733704 16946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=36985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:22.734941 16946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:22.736248 16946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:22.748368 16951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:22.748426 16952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:22.748446 16954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:22.749976 16946 server_base.cc:1034] running on GCE node
I20260924 19:04:22.750651 16946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:22.751974 16946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:22.753196 16946 hybrid_clock.cc:648] HybridClock initialized: now 1790276662753122 us; error 40 us; skew 500 ppm
I20260924 19:04:22.753590 16946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:04:22 dist-test-slave-9fn1 krb5kdc[15864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:04:22.756611 16946 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:04:22.758092 16946 webserver.cc:460] Webserver started at http://127.0.97.62:36985/ using document root <none> and password file <none>
I20260924 19:04:22.758816 16946 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:22.758946 16946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:22.764853 16946 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 19:04:22.767731 16961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:22.769186 16946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:04:22.769376 16946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "276693ed963e44a6827ffc8d97d8356f"
format_stamp: "Formatted at 2026-09-24 19:03:36 on dist-test-slave-9fn1"
server_key: "a8c1ce2fe86d58c058774f59c89fa6fd"
server_key_iv: "55309b4e33838f818092f9bca0d7d690"
server_key_version: "encryptionkey@0"
I20260924 19:04:22.770383 16946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:22.788414 16946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:22.792704 16946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:22.792974 16946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:22.810550 16946 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42697
I20260924 19:04:22.810595 17012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42697 every 8 connection(s)
I20260924 19:04:22.812024 16946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:04:22.818085 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 16946
I20260924 19:04:22.818924 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:44949
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:42697
--webserver_port=42257
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:04:22.819166 17013 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:04:22.822431 17013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f: Bootstrap starting.
I20260924 19:04:22.851294 17013 log.cc:824] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f: Log is configured to *not* fsync() on all Append() calls
I20260924 19:04:22.871593 17013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:04:22.872241 17013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f: Bootstrap complete.
I20260924 19:04:22.884727 17013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276693ed963e44a6827ffc8d97d8356f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42697 } }
I20260924 19:04:22.885684 17013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 276693ed963e44a6827ffc8d97d8356f, State: Initialized, Role: FOLLOWER
I20260924 19:04:22.886325 17013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276693ed963e44a6827ffc8d97d8356f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42697 } }
I20260924 19:04:22.886588 17013 raft_consensus.cc:388] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:04:22.886715 17013 raft_consensus.cc:482] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:04:22.886924 17013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [term 2 FOLLOWER]: Advancing to term 3
I20260924 19:04:22.890926 17013 raft_consensus.cc:504] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276693ed963e44a6827ffc8d97d8356f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42697 } }
I20260924 19:04:22.891423 17013 leader_election.cc:302] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 276693ed963e44a6827ffc8d97d8356f; no voters:
I20260924 19:04:22.892084 17013 leader_election.cc:288] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 19:04:22.892271 17017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [term 3 FOLLOWER]: Leader election won for term 3
I20260924 19:04:22.893425 17017 raft_consensus.cc:686] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [term 3 LEADER]: Becoming Leader. State: Replica: 276693ed963e44a6827ffc8d97d8356f, State: Running, Role: LEADER
I20260924 19:04:22.894151 17013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:04:22.894068 17017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276693ed963e44a6827ffc8d97d8356f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42697 } }
I20260924 19:04:22.900673 17018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "276693ed963e44a6827ffc8d97d8356f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276693ed963e44a6827ffc8d97d8356f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42697 } } }
I20260924 19:04:22.901021 17018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [sys.catalog]: This master's current role is: LEADER
I20260924 19:04:22.901762 17019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 276693ed963e44a6827ffc8d97d8356f. Latest consensus state: current_term: 3 leader_uuid: "276693ed963e44a6827ffc8d97d8356f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276693ed963e44a6827ffc8d97d8356f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42697 } } }
I20260924 19:04:22.902128 17019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f [sys.catalog]: This master's current role is: LEADER
I20260924 19:04:22.916610 17031 catalog_manager.cc:1259] Loaded cluster ID: 71d271b6a53a4395bc084281617a1158
I20260924 19:04:22.916741 17031 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f: loading cluster ID for follower catalog manager: success
I20260924 19:04:22.920964 17031 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 276693ed963e44a6827ffc8d97d8356f: acquiring CA information for follower catalog manager: success
F20260924 19:04:22.923309 17031 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f831998e9ad google::LogMessage::Fail() at ??:0
@ 0x7f831999188a google::LogMessage::SendToLog() at ??:0
@ 0x7f831998e4de google::LogMessage::Flush() at ??:0
@ 0x7f831998fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f831d6f27d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f832f3c3bbf kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f832f3c317b kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f832f3b2247 kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f832f3abc56 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f832f44600c kudu::master::CatalogManagerBgTasks::Init()::$_0::operator()() at ??:0
@ 0x7f832f445e89 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f831c5b8eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f831f8926db start_thread at ??:0
@ 0x7f831882861f clone at ??:0
W20260924 19:04:23.209149 17015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:23.209407 17015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:23.209542 17015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:23.216706 17015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:23.216807 17015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:23.216931 17015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:04:23.226452 17015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:44949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=42257
--tserver_master_addrs=127.0.97.62:42697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:23.227583 17015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:23.228803 17015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:23.241822 17054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:23.241927 17053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:23.241945 17056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:23.242888 17015 server_base.cc:1034] running on GCE node
I20260924 19:04:23.243458 17015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:23.244341 17015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:23.245599 17015 hybrid_clock.cc:648] HybridClock initialized: now 1790276663245551 us; error 38 us; skew 500 ppm
I20260924 19:04:23.245919 17015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:04:23 dist-test-slave-9fn1 krb5kdc[15864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:04:23.248974 17015 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 19:04:23.250443 17015 webserver.cc:460] Webserver started at http://127.0.97.1:42257/ using document root <none> and password file <none>
I20260924 19:04:23.251132 17015 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:23.251278 17015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:23.256671 17015 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.003s
I20260924 19:04:23.259395 17063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:23.260666 17015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260924 19:04:23.260831 17015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "5c688e4544644d3aa51249bdd95e0485"
format_stamp: "Formatted at 2026-09-24 19:03:37 on dist-test-slave-9fn1"
server_key: "b7a4105efd9ae35f9d7a5fc0d3e0f65b"
server_key_iv: "2b8b963ddd332d35098011b4d0a70d3c"
server_key_version: "encryptionkey@0"
I20260924 19:04:23.261847 17015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:23.275818 17015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:23.279546 17015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:23.279814 17015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:23.280695 17015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:04:23.281982 17015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:04:23.282107 17015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:23.282241 17015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:04:23.282331 17015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:23.315866 17015 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44949
I20260924 19:04:23.315917 17175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44949 every 8 connection(s)
I20260924 19:04:23.317291 17015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:04:23.322422 17178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:23.320463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.1:47377)
0924 19:04:23.321206 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:23.321427 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":292,"threads_started":1}
W20260924 19:04:23.323127 17176 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42697 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:42697: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
I20260924 19:04:23.323982 17178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:23.323656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.1:51981)
0924 19:04:23.323802 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:23.323870 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 19:04:23.324839 17178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:23.324505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.1:41499)
0924 19:04:23.324645 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:23.324720 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 19:04:23.325203 17176 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 19:04:23.326141 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 17015
I20260924 19:04:23.326829 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:41921
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:42697
--webserver_port=43137
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:04:23.650612 17179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:23.650925 17179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:23.651042 17179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:23.658097 17179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:23.658193 17179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:23.658308 17179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:04:23.668212 17179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:41921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=43137
--tserver_master_addrs=127.0.97.62:42697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:23.669358 17179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:23.670548 17179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:23.683250 17184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:23.683250 17185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:23.683286 17187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:23.684077 17179 server_base.cc:1034] running on GCE node
I20260924 19:04:23.684587 17179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:23.685465 17179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:23.686729 17179 hybrid_clock.cc:648] HybridClock initialized: now 1790276663686688 us; error 34 us; skew 500 ppm
I20260924 19:04:23.687023 17179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:04:23 dist-test-slave-9fn1 krb5kdc[15864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:04:23.696094 17179 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 19:04:23.697470 17179 webserver.cc:460] Webserver started at http://127.0.97.2:43137/ using document root <none> and password file <none>
I20260924 19:04:23.698127 17179 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:23.698230 17179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:23.703290 17179 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:04:23.705844 17194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:23.707115 17179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 19:04:23.707265 17179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "54662ec87ba34e70a719d89f450b8d1e"
format_stamp: "Formatted at 2026-09-24 19:03:38 on dist-test-slave-9fn1"
server_key: "c4099ecfadc935f586bd0166104dd6e1"
server_key_iv: "0972755de12e7748cde063fc253c9e6f"
server_key_version: "encryptionkey@0"
I20260924 19:04:23.708173 17179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:23.722700 17179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:23.726765 17179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:23.726975 17179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:23.727813 17179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:04:23.728969 17179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:04:23.729061 17179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:23.729136 17179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:04:23.729193 17179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:23.762092 17179 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41921
I20260924 19:04:23.762140 17306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41921 every 8 connection(s)
I20260924 19:04:23.763512 17179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:04:23.768247 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:23.766514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.2:37315)
0924 19:04:23.767218 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:23.767404 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":272,"threads_started":1}
W20260924 19:04:23.768851 17307 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42697 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:42697: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
I20260924 19:04:23.769605 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:23.769247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.2:42019)
0924 19:04:23.769378 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:23.769442 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 19:04:23.770290 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 17179
I20260924 19:04:23.770509 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:23.770113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.2:39117)
0924 19:04:23.770262 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:23.770378 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 19:04:23.770753 17307 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 19:04:23.771026 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:44109
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:42697
--webserver_port=43851
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:04:24.088169 17310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:24.088492 17310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:24.088666 17310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:24.095947 17310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:24.096099 17310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:24.096268 17310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:04:24.105836 17310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:44109
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43851
--tserver_master_addrs=127.0.97.62:42697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:24.107092 17310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:24.108381 17310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:24.122061 17315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:24.122146 17316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:24.122109 17318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:24.122886 17310 server_base.cc:1034] running on GCE node
I20260924 19:04:24.123447 17310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:24.124315 17310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:24.125584 17310 hybrid_clock.cc:648] HybridClock initialized: now 1790276664125526 us; error 59 us; skew 500 ppm
I20260924 19:04:24.125892 17310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:04:24 dist-test-slave-9fn1 krb5kdc[15864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:04:24.128872 17310 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 19:04:24.130352 17310 webserver.cc:460] Webserver started at http://127.0.97.3:43851/ using document root <none> and password file <none>
I20260924 19:04:24.130995 17310 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:24.131151 17310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:24.136348 17310 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 19:04:24.139036 17325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:24.140338 17310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:04:24.140502 17310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "4db698cbb4b045b98841a62ac7a61b49"
format_stamp: "Formatted at 2026-09-24 19:03:38 on dist-test-slave-9fn1"
server_key: "3e71cc6f3d136d932d619dd1ec3b42bc"
server_key_iv: "1817b158ce5782877eade30f2c36cf4c"
server_key_version: "encryptionkey@0"
I20260924 19:04:24.141430 17310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:24.166591 17310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:24.170506 17310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:24.170791 17310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:24.171890 17310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:04:24.173156 17310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:04:24.173274 17310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:24.173399 17310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:04:24.173487 17310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:24.206724 17310 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44109
I20260924 19:04:24.206812 17437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44109 every 8 connection(s)
I20260924 19:04:24.208169 17310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:04:24.213073 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 17310
I20260924 19:04:24.213147 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:24.211279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.3:36399)
0924 19:04:24.212010 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:24.212289 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":263,"threads_started":1}
W20260924 19:04:24.213800 17438 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42697 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:42697: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
I20260924 19:04:24.214560 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:24.214249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.3:57269)
0924 19:04:24.214391 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:24.214468 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 19:04:24.215456 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:24.215148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.3:41907)
0924 19:04:24.215274 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:24.215346 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 19:04:24.215724 17438 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 19:04:24.326480 17442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:24.325744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.1:50863)
0924 19:04:24.326166 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:24.326248 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":197,"threads_started":1}
I20260924 19:04:24.772020 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:24.771305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.2:55767)
0924 19:04:24.771725 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:24.771807 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":201,"threads_started":1}
I20260924 19:04:25.216934 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:25.216205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.3:44289)
0924 19:04:25.216635 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:25.216716 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":217,"threads_started":1}
I20260924 19:04:25.328018 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:25.327325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.1:47601)
0924 19:04:25.327776 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:25.327858 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":238,"threads_started":1}
I20260924 19:04:25.773798 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:25.773036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.2:33221)
0924 19:04:25.773486 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:25.773638 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":233,"threads_started":1}
I20260924 19:04:26.218637 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:26.217803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.3:42429)
0924 19:04:26.218376 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:26.218479 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":320,"threads_started":1}
I20260924 19:04:26.329687 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:26.328808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.1:47755)
0924 19:04:26.329378 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:26.329456 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":362,"threads_started":1}
I20260924 19:04:26.775243 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:26.774582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.2:51569)
0924 19:04:26.775010 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:26.775084 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":233,"threads_started":1}
I20260924 19:04:27.220088 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:27.219450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.3:59757)
0924 19:04:27.219860 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:27.219932 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":225,"threads_started":1}
I20260924 19:04:27.331255 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:27.330570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.1:57867)
0924 19:04:27.330984 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:27.331056 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":207,"threads_started":1}
I20260924 19:04:27.776923 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:27.776196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.2:43977)
0924 19:04:27.776641 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:27.776725 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":215,"threads_started":1}
I20260924 19:04:28.221721 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:28.220999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.3:43215)
0924 19:04:28.221390 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:28.221463 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":198,"threads_started":1}
I20260924 19:04:28.332793 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:28.332111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.1:41355)
0924 19:04:28.332548 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:28.332630 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":223,"threads_started":1}
I20260924 19:04:28.778810 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:28.777926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.2:58453)
0924 19:04:28.778474 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:28.778589 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":312,"threads_started":1}
I20260924 19:04:29.223281 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:29.222608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.3:33843)
0924 19:04:29.223048 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:29.223122 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":221,"threads_started":1}
I20260924 19:04:29.334293 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:29.333622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.1:52161)
0924 19:04:29.334048 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:29.334131 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":225,"threads_started":1}
I20260924 19:04:29.780311 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:29.779634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.2:33817)
0924 19:04:29.780072 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:29.780157 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":241,"threads_started":1}
I20260924 19:04:30.224823 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:30.224114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.3:39181)
0924 19:04:30.224538 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:30.224654 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":236,"threads_started":1}
I20260924 19:04:30.336064 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:30.335291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.1:32973)
0924 19:04:30.335777 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:30.335854 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":229,"threads_started":1}
I20260924 19:04:30.782069 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:30.781262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.2:36581)
0924 19:04:30.781778 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:30.781856 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":279,"threads_started":1}
I20260924 19:04:31.226627 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:31.225828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.3:54745)
0924 19:04:31.226339 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:31.226422 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":234,"threads_started":1}
I20260924 19:04:31.337905 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:31.337128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.1:42421)
0924 19:04:31.337660 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:31.337745 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":250,"threads_started":1}
I20260924 19:04:31.783895 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:31.783153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.2:57239)
0924 19:04:31.783652 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:31.783738 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":264,"threads_started":1}
I20260924 19:04:32.228404 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:32.227688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.3:52957)
0924 19:04:32.228179 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:32.228250 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":294,"threads_started":1}
I20260924 19:04:32.339378 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:32.338702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.1:33789)
0924 19:04:32.339142 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:32.339214 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":219,"threads_started":1}
I20260924 19:04:32.785630 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:32.784820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.2:36813)
0924 19:04:32.785319 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:32.785396 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":241,"threads_started":1}
I20260924 19:04:33.229995 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:33.229232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.3:48707)
0924 19:04:33.229749 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:33.229828 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":298,"threads_started":1}
I20260924 19:04:33.340989 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:33.340277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.1:33847)
0924 19:04:33.340740 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:33.340821 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":253,"threads_started":1}
I20260924 19:04:33.787153 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:33.786500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.2:55903)
0924 19:04:33.786920 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:33.786997 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":230,"threads_started":1}
I20260924 19:04:34.231576 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:34.230892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.3:32807)
0924 19:04:34.231337 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:34.231423 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":230,"threads_started":1}
I20260924 19:04:34.342638 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:34.341928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.1:53963)
0924 19:04:34.342362 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:34.342460 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":224,"threads_started":1}
I20260924 19:04:34.789004 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:34.788280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.2:44787)
0924 19:04:34.788735 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:34.788830 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":238,"threads_started":1}
I20260924 19:04:35.233275 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:35.232582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.3:58591)
0924 19:04:35.232996 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:35.233087 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":193,"threads_started":1}
I20260924 19:04:35.344249 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:35.343557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.1:55649)
0924 19:04:35.344016 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:35.344097 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":236,"threads_started":1}
I20260924 19:04:35.790916 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:35.790190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.2:47307)
0924 19:04:35.790668 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:35.790757 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":253,"threads_started":1}
I20260924 19:04:36.235024 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:36.234329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.3:56505)
0924 19:04:36.234777 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:36.234867 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":238,"threads_started":1}
I20260924 19:04:36.346230 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:36.345279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.1:47507)
0924 19:04:36.345851 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:36.345964 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":280,"threads_started":1}
I20260924 19:04:36.792611 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:36.791905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.2:58829)
0924 19:04:36.792339 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:36.792421 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":216,"threads_started":1}
I20260924 19:04:37.236701 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:37.235977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.3:33223)
0924 19:04:37.236446 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:37.236526 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":223,"threads_started":1}
I20260924 19:04:37.347762 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:37.347065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.1:58817)
0924 19:04:37.347476 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:37.347549 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":218,"threads_started":1}
I20260924 19:04:37.794292 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:37.793596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.2:56311)
0924 19:04:37.794027 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:37.794116 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":213,"threads_started":1}
I20260924 19:04:38.238315 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:38.237686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.3:35481)
0924 19:04:38.238081 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:38.238149 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":196,"threads_started":1}
I20260924 19:04:38.349578 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:38.348850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.1:60189)
0924 19:04:38.349278 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:38.349363 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":224,"threads_started":1}
I20260924 19:04:38.796291 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:38.795504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.2:45553)
0924 19:04:38.796008 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:38.796110 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":275,"threads_started":1}
I20260924 19:04:39.220840 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 17015
I20260924 19:04:39.238771 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:39.222216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55768 (local address 127.0.97.1:44949)
0924 19:04:39.223076 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:39.223090 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:39.223128 (+ 38us) server_negotiation.cc:407] Connection header received
0924 19:04:39.224616 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:39.224625 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:39.225048 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:39.225345 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:39.226759 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:39.228056 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:39.229261 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:39.230016 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:39.230237 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:39.232816 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:39.232824 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:39.232856 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:39.232905 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:39.235903 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:39.236504 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:39.236511 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:39.236517 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:39.236677 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:39.236991 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:39.236994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:39.236996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:39.237482 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:39.237775 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:39.238061 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 19:04:39.238498 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":576,"threads_started":1}
I20260924 19:04:39.240111 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:39.239376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42697 (local address 127.0.97.3:57497)
0924 19:04:39.239842 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:39.239920 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:42697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":271,"threads_started":1}
I20260924 19:04:39.240483 17171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:04:39.323203 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 17015
I20260924 19:04:39.340484 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 17179
I20260924 19:04:39.358129 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:39.341394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40072 (local address 127.0.97.2:41921)
0924 19:04:39.341970 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:39.341985 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:39.342025 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:04:39.343542 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:39.343550 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:39.343975 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:39.344328 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:39.345999 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:39.347188 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:39.348492 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:39.349329 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:39.349563 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:39.351989 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:39.351997 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:39.352014 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:39.352049 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:39.354911 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:39.355566 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:39.355573 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:39.355580 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:39.355732 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:39.356130 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:39.356134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:39.356136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:39.356636 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:39.356931 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:39.357411 (+ 480us) server_negotiation.cc:299] Negotiation successful
0924 19:04:39.357831 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":297,"threads_started":1}
I20260924 19:04:39.359467 17302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:04:39.442750 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 17179
I20260924 19:04:39.460391 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 17310
I20260924 19:04:39.477841 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:39.461385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45724 (local address 127.0.97.3:44109)
0924 19:04:39.462038 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:39.462052 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:39.462091 (+ 39us) server_negotiation.cc:407] Connection header received
0924 19:04:39.463694 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:39.463703 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:39.464145 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:39.464514 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:39.466047 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:39.467264 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:39.468525 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:39.469184 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:39.469387 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:39.471970 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:39.471977 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:39.471992 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:39.472037 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:39.474958 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:39.475558 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:39.475563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:39.475569 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:39.475735 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:39.476073 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:39.476076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:39.476078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:39.476544 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:39.476769 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:39.477080 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 19:04:39.477493 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":358,"threads_started":1}
I20260924 19:04:39.479209 17433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:04:39.561461 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 17310
2026-09-24T19:04:39Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64271 ms)
[----------] 30 tests from SecurityITest (292958 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:04:39 dist-test-slave-9fn1 krb5kdc[17495](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:04:39 dist-test-slave-9fn1 krb5kdc[17495](info): set up 2 sockets
Sep 24 19:04:39 dist-test-slave-9fn1 krb5kdc[17495](info): commencing operation
krb5kdc: starting...
W20260924 19:04:40.685930 388 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:04:40 dist-test-slave-9fn1 krb5kdc[17495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:04:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:04:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 19:04:40.924875 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40047
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:04:41.240732 17508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:41.241010 17508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:41.241163 17508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:41.248311 17508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:04:41.248461 17508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:41.248545 17508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:41.248651 17508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:04:41.248716 17508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:04:41.258694 17508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:41.259948 17508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:41.261163 17508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:41.272447 17513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:41.272486 17514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:41.272704 17516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:41.273350 17508 server_base.cc:1034] running on GCE node
I20260924 19:04:41.274143 17508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:41.275486 17508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:41.276687 17508 hybrid_clock.cc:648] HybridClock initialized: now 1790276681276652 us; error 29 us; skew 500 ppm
I20260924 19:04:41.276975 17508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:04:41 dist-test-slave-9fn1 krb5kdc[17495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:04:41.279945 17508 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:04:41.281255 17508 webserver.cc:460] Webserver started at http://127.0.97.62:40507/ using document root <none> and password file <none>
I20260924 19:04:41.281965 17508 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:41.282091 17508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:41.282406 17508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:04:41.285238 17508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "24fb55e0a9da4f1bb881d825d851c962"
format_stamp: "Formatted at 2026-09-24 19:04:41 on dist-test-slave-9fn1"
server_key: "e38d90d5bb64d6fe56ac080099c6d6e0"
server_key_iv: "890e7fff22aa5d662043f8287d79c8bf"
server_key_version: "encryptionkey@0"
I20260924 19:04:41.286099 17508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "24fb55e0a9da4f1bb881d825d851c962"
format_stamp: "Formatted at 2026-09-24 19:04:41 on dist-test-slave-9fn1"
server_key: "e38d90d5bb64d6fe56ac080099c6d6e0"
server_key_iv: "890e7fff22aa5d662043f8287d79c8bf"
server_key_version: "encryptionkey@0"
I20260924 19:04:41.290822 17508 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 19:04:41.293912 17523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:41.295133 17508 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 19:04:41.295305 17508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "24fb55e0a9da4f1bb881d825d851c962"
format_stamp: "Formatted at 2026-09-24 19:04:41 on dist-test-slave-9fn1"
server_key: "e38d90d5bb64d6fe56ac080099c6d6e0"
server_key_iv: "890e7fff22aa5d662043f8287d79c8bf"
server_key_version: "encryptionkey@0"
I20260924 19:04:41.295508 17508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:41.318782 17508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:41.322551 17508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:41.322846 17508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:41.340628 17508 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40047
I20260924 19:04:41.340667 17574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40047 every 8 connection(s)
I20260924 19:04:41.341931 17508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:04:41.345618 17575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:04:41.345870 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 17508
I20260924 19:04:41.346112 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:04:41.346957 388 external_mini_cluster.cc:1428] Setting key c9a7baff914efcd47c86222ab3ecfcca
I20260924 19:04:41.359455 17575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24fb55e0a9da4f1bb881d825d851c962: Bootstrap starting.
Sep 24 19:04:41 dist-test-slave-9fn1 krb5kdc[17495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:04:41.363014 17575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24fb55e0a9da4f1bb881d825d851c962: Neither blocks nor log segments found. Creating new log.
I20260924 19:04:41.364033 17575 log.cc:824] T 00000000000000000000000000000000 P 24fb55e0a9da4f1bb881d825d851c962: Log is configured to *not* fsync() on all Append() calls
I20260924 19:04:41.366859 17575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24fb55e0a9da4f1bb881d825d851c962: No bootstrap required, opened a new log
I20260924 19:04:41.370455 17578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:41.349017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45128 (local address 127.0.97.62:40047)
0924 19:04:41.349767 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:41.349778 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:41.349820 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:04:41.352013 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:41.352036 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:41.352444 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:41.352783 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:41.354507 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:41.355767 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:41.357731 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:41.358376 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:41.358501 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:41.363060 (+ 4559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:41.363067 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:41.363079 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:41.363112 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:41.366473 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:41.367135 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:41.367141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:41.367147 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:41.367326 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:41.367728 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:41.367739 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:41.367744 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:41.368242 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:41.368590 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:41.369032 (+ 442us) server_negotiation.cc:299] Negotiation successful
0924 19:04:41.369442 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":350,"threads_started":1}
I20260924 19:04:41.374800 17575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24fb55e0a9da4f1bb881d825d851c962 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fb55e0a9da4f1bb881d825d851c962" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40047 } }
I20260924 19:04:41.375165 17575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24fb55e0a9da4f1bb881d825d851c962 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:04:41.375272 17575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24fb55e0a9da4f1bb881d825d851c962 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24fb55e0a9da4f1bb881d825d851c962, State: Initialized, Role: FOLLOWER
I20260924 19:04:41.375851 17575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24fb55e0a9da4f1bb881d825d851c962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fb55e0a9da4f1bb881d825d851c962" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40047 } }
I20260924 19:04:41.376098 17575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24fb55e0a9da4f1bb881d825d851c962 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:04:41.376246 17575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24fb55e0a9da4f1bb881d825d851c962 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:04:41.376425 17575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24fb55e0a9da4f1bb881d825d851c962 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:04:41.378753 17575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24fb55e0a9da4f1bb881d825d851c962 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fb55e0a9da4f1bb881d825d851c962" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40047 } }
I20260924 19:04:41.379300 17575 leader_election.cc:302] T 00000000000000000000000000000000 P 24fb55e0a9da4f1bb881d825d851c962 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24fb55e0a9da4f1bb881d825d851c962; no voters:
I20260924 19:04:41.379916 17575 leader_election.cc:288] T 00000000000000000000000000000000 P 24fb55e0a9da4f1bb881d825d851c962 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:04:41.380007 17580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24fb55e0a9da4f1bb881d825d851c962 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:04:41.380745 17580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24fb55e0a9da4f1bb881d825d851c962 [term 1 LEADER]: Becoming Leader. State: Replica: 24fb55e0a9da4f1bb881d825d851c962, State: Running, Role: LEADER
I20260924 19:04:41.381304 17580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24fb55e0a9da4f1bb881d825d851c962 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fb55e0a9da4f1bb881d825d851c962" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40047 } }
I20260924 19:04:41.382027 17575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24fb55e0a9da4f1bb881d825d851c962 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:04:41.384200 17582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24fb55e0a9da4f1bb881d825d851c962 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24fb55e0a9da4f1bb881d825d851c962. Latest consensus state: current_term: 1 leader_uuid: "24fb55e0a9da4f1bb881d825d851c962" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fb55e0a9da4f1bb881d825d851c962" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40047 } } }
I20260924 19:04:41.384621 17582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24fb55e0a9da4f1bb881d825d851c962 [sys.catalog]: This master's current role is: LEADER
I20260924 19:04:41.384529 17581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24fb55e0a9da4f1bb881d825d851c962 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24fb55e0a9da4f1bb881d825d851c962" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24fb55e0a9da4f1bb881d825d851c962" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40047 } } }
I20260924 19:04:41.384814 17581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24fb55e0a9da4f1bb881d825d851c962 [sys.catalog]: This master's current role is: LEADER
I20260924 19:04:41.387044 17588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:04:41.391556 17588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:04:41.400870 17588 catalog_manager.cc:1347] Generated new cluster ID: f23846efc1484c5f9b52ae9c7231e43f
I20260924 19:04:41.400975 17588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:04:41.416780 17588 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:04:41.418673 17588 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:04:41.431635 17588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24fb55e0a9da4f1bb881d825d851c962: Generated new TSK 0
I20260924 19:04:41.432595 17588 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 19:04:41.488942 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40047
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:04:41.808266 17601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:41.808550 17601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:41.808665 17601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:41.815707 17601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:41.815804 17601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:41.815912 17601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:04:41.825002 17601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:41.826366 17601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:41.827577 17601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:41.839883 17606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:41.839968 17609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:41.839926 17607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:41.840786 17601 server_base.cc:1034] running on GCE node
I20260924 19:04:41.841354 17601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:41.842299 17601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:41.843524 17601 hybrid_clock.cc:648] HybridClock initialized: now 1790276681843489 us; error 46 us; skew 500 ppm
I20260924 19:04:41.843823 17601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:04:41 dist-test-slave-9fn1 krb5kdc[17495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:04:41.846832 17601 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 19:04:41.848178 17601 webserver.cc:460] Webserver started at http://127.0.97.1:40577/ using document root <none> and password file <none>
I20260924 19:04:41.848850 17601 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:41.849002 17601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:41.849303 17601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:04:41.852097 17601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-0/data/instance:
uuid: "50d35e55a23943a59817d2794dce4ff7"
format_stamp: "Formatted at 2026-09-24 19:04:41 on dist-test-slave-9fn1"
server_key: "98ce5732785769d97df5cbbfe3aa9e93"
server_key_iv: "78896870727e9986efc8d085a622f1c2"
server_key_version: "encryptionkey@0"
I20260924 19:04:41.852859 17601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-0/wal/instance:
uuid: "50d35e55a23943a59817d2794dce4ff7"
format_stamp: "Formatted at 2026-09-24 19:04:41 on dist-test-slave-9fn1"
server_key: "98ce5732785769d97df5cbbfe3aa9e93"
server_key_iv: "78896870727e9986efc8d085a622f1c2"
server_key_version: "encryptionkey@0"
I20260924 19:04:41.857750 17601 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260924 19:04:41.860946 17616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:41.862164 17601 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 19:04:41.862339 17601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "50d35e55a23943a59817d2794dce4ff7"
format_stamp: "Formatted at 2026-09-24 19:04:41 on dist-test-slave-9fn1"
server_key: "98ce5732785769d97df5cbbfe3aa9e93"
server_key_iv: "78896870727e9986efc8d085a622f1c2"
server_key_version: "encryptionkey@0"
I20260924 19:04:41.862545 17601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:41.885982 17601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:41.889986 17601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:41.890273 17601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:41.891175 17601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:04:41.892408 17601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:04:41.892479 17601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:41.892597 17601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:04:41.892657 17601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:41.925362 17601 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46027
I20260924 19:04:41.925419 17728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46027 every 8 connection(s)
I20260924 19:04:41.926635 17601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:04:41.931749 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 17601
I20260924 19:04:41.931939 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-0/wal/instance
I20260924 19:04:41.932750 388 external_mini_cluster.cc:1428] Setting key b2e47d18527d43f357dfe195c980b4b9
Sep 24 19:04:41 dist-test-slave-9fn1 krb5kdc[17495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:04:41.948338 17731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:41.929010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34919 (local address 127.0.97.62:40047)
0924 19:04:41.929447 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:41.929453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:41.930416 (+ 963us) server_negotiation.cc:407] Connection header received
0924 19:04:41.932746 (+ 2330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:41.932758 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:41.932849 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:41.932950 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:41.934681 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:41.935415 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:41.937910 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:41.938296 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:41.938388 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:41.942425 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:41.942430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:41.942433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:41.942463 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:41.945016 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:41.945816 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:41.945825 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:41.945831 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:41.945985 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:41.946387 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:41.946394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:41.946401 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:41.946780 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:41.946931 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:41.947616 (+ 685us) server_negotiation.cc:299] Negotiation successful
0924 19:04:41.947869 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":198,"threads_started":1}
I20260924 19:04:41.948894 17732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:41.929587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40047 (local address 127.0.97.1:34919)
0924 19:04:41.930263 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:41.930306 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:41.932432 (+ 2126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:41.933207 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:41.933217 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:41.933832 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:04:41.934532 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:41.934550 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:41.935559 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:41.935567 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:41.937760 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:41.937778 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:41.938516 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:41.938525 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:41.938828 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:41.939548 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:04:41.939581 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:04:41.942196 (+ 2615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:04:41.945147 (+ 2951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:04:41.945153 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:04:41.945165 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:04:41.945683 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:04:41.946092 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:04:41.946095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:04:41.946097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:04:41.946272 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:04:41.946904 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:04:41.946910 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:04:41.947211 (+ 301us) client_negotiation.cc:769] Sending connection context
0924 19:04:41.947440 (+ 229us) client_negotiation.cc:240] Negotiation successful
0924 19:04:41.947813 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":282,"threads_started":1}
I20260924 19:04:41.950423 17729 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40047
I20260924 19:04:41.950721 17729 heartbeater.cc:464] Registering TS with master...
I20260924 19:04:41.951411 17729 heartbeater.cc:511] Master 127.0.97.62:40047 requested a full tablet report, sending...
I20260924 19:04:41.953634 17540 ts_manager.cc:194] Registered new tserver with Master: 50d35e55a23943a59817d2794dce4ff7 (127.0.97.1:46027)
I20260924 19:04:41.955850 17540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34919
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 19:04:41.988363 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40047
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:04:42.303580 17735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:42.303882 17735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:42.304054 17735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:42.311132 17735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:42.311278 17735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:42.311468 17735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:04:42.320947 17735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:42.322217 17735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:42.323454 17735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:42.336241 17740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:42.336221 17741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:42.336385 17743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:42.336998 17735 server_base.cc:1034] running on GCE node
I20260924 19:04:42.337648 17735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:42.338523 17735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:42.339715 17735 hybrid_clock.cc:648] HybridClock initialized: now 1790276682339653 us; error 51 us; skew 500 ppm
I20260924 19:04:42.340036 17735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:04:42 dist-test-slave-9fn1 krb5kdc[17495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:04:42.343255 17735 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 19:04:42.344671 17735 webserver.cc:460] Webserver started at http://127.0.97.2:45941/ using document root <none> and password file <none>
I20260924 19:04:42.345319 17735 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:42.345438 17735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:42.345786 17735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:04:42.348489 17735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-1/data/instance:
uuid: "b711b03d0dec4a51b717e6f99a1356bc"
format_stamp: "Formatted at 2026-09-24 19:04:42 on dist-test-slave-9fn1"
server_key: "d68c6cdb5bdc540c61737e905adfcb2c"
server_key_iv: "9153623f3a0079f66ed3137500148f91"
server_key_version: "encryptionkey@0"
I20260924 19:04:42.349248 17735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b711b03d0dec4a51b717e6f99a1356bc"
format_stamp: "Formatted at 2026-09-24 19:04:42 on dist-test-slave-9fn1"
server_key: "d68c6cdb5bdc540c61737e905adfcb2c"
server_key_iv: "9153623f3a0079f66ed3137500148f91"
server_key_version: "encryptionkey@0"
I20260924 19:04:42.354151 17735 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 19:04:42.357383 17750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:42.358693 17735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:04:42.358852 17735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "b711b03d0dec4a51b717e6f99a1356bc"
format_stamp: "Formatted at 2026-09-24 19:04:42 on dist-test-slave-9fn1"
server_key: "d68c6cdb5bdc540c61737e905adfcb2c"
server_key_iv: "9153623f3a0079f66ed3137500148f91"
server_key_version: "encryptionkey@0"
I20260924 19:04:42.359046 17735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:42.371153 17735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:42.375388 17735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:42.375689 17735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:42.376567 17735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:04:42.377873 17735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:04:42.378006 17735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:42.378159 17735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:04:42.378237 17735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:42.410770 17735 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40157
I20260924 19:04:42.410805 17862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40157 every 8 connection(s)
I20260924 19:04:42.412014 17735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:04:42.419296 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 17735
I20260924 19:04:42.419512 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-1/wal/instance
I20260924 19:04:42.420044 388 external_mini_cluster.cc:1428] Setting key fca646f171f67e264b5954ba70f5e106
Sep 24 19:04:42 dist-test-slave-9fn1 krb5kdc[17495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:04:42.434095 17731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:42.414566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35635 (local address 127.0.97.62:40047)
0924 19:04:42.414743 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:42.414748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:42.415907 (+ 1159us) server_negotiation.cc:407] Connection header received
0924 19:04:42.417767 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:42.417773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:42.417861 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:42.417970 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:42.419701 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:42.420747 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:42.422771 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:42.423492 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:42.423631 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:42.426934 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:42.426952 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:42.426960 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:42.427014 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:42.430483 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:42.431452 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:42.431460 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:42.431466 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:42.431629 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:42.432151 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:42.432159 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:42.432165 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:42.432552 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:42.432700 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:42.433332 (+ 632us) server_negotiation.cc:299] Negotiation successful
0924 19:04:42.433639 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 19:04:42.434711 17865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:42.415014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40047 (local address 127.0.97.2:35635)
0924 19:04:42.415758 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:42.415796 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:42.417444 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:42.418206 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:42.418214 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:42.418694 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:04:42.419463 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:42.419508 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:42.420918 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:42.420922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:42.422620 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:42.422631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:42.423700 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:42.423704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:42.423891 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:42.424562 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:04:42.424584 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:04:42.426715 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:04:42.430632 (+ 3917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:04:42.430643 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:04:42.430670 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:04:42.431309 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:04:42.431766 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:04:42.431770 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:04:42.431773 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:04:42.432013 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:04:42.432672 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:04:42.432678 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:04:42.432977 (+ 299us) client_negotiation.cc:769] Sending connection context
0924 19:04:42.433190 (+ 213us) client_negotiation.cc:240] Negotiation successful
0924 19:04:42.433627 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":295,"threads_started":1}
I20260924 19:04:42.436209 17863 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40047
I20260924 19:04:42.436534 17863 heartbeater.cc:464] Registering TS with master...
I20260924 19:04:42.437254 17863 heartbeater.cc:511] Master 127.0.97.62:40047 requested a full tablet report, sending...
I20260924 19:04:42.438777 17540 ts_manager.cc:194] Registered new tserver with Master: b711b03d0dec4a51b717e6f99a1356bc (127.0.97.2:40157)
I20260924 19:04:42.439555 17540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35635
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 19:04:42.476104 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40047
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:04:42.798785 17868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:42.799105 17868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:42.799288 17868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:42.806448 17868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:42.806569 17868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:42.806756 17868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:04:42.816030 17868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40047
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:42.817297 17868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:42.818629 17868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:42.831049 17873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:42.831260 17874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:42.831527 17876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:42.832154 17868 server_base.cc:1034] running on GCE node
I20260924 19:04:42.832723 17868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:42.833622 17868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:42.834831 17868 hybrid_clock.cc:648] HybridClock initialized: now 1790276682834788 us; error 46 us; skew 500 ppm
I20260924 19:04:42.835146 17868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:04:42 dist-test-slave-9fn1 krb5kdc[17495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:04:42.838153 17868 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 19:04:42.839565 17868 webserver.cc:460] Webserver started at http://127.0.97.3:45953/ using document root <none> and password file <none>
I20260924 19:04:42.840252 17868 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:42.840392 17868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:42.840688 17868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:04:42.843552 17868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-2/data/instance:
uuid: "bca3648be64f4e01b9ec93727c0d5601"
format_stamp: "Formatted at 2026-09-24 19:04:42 on dist-test-slave-9fn1"
server_key: "ffec685851b9b79ee755981d67c54e72"
server_key_iv: "057c16ca6b63b5e6c81e910dcc6c40e2"
server_key_version: "encryptionkey@0"
I20260924 19:04:42.844322 17868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-2/wal/instance:
uuid: "bca3648be64f4e01b9ec93727c0d5601"
format_stamp: "Formatted at 2026-09-24 19:04:42 on dist-test-slave-9fn1"
server_key: "ffec685851b9b79ee755981d67c54e72"
server_key_iv: "057c16ca6b63b5e6c81e910dcc6c40e2"
server_key_version: "encryptionkey@0"
I20260924 19:04:42.849117 17868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260924 19:04:42.852434 17883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:42.853729 17868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:04:42.853892 17868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "bca3648be64f4e01b9ec93727c0d5601"
format_stamp: "Formatted at 2026-09-24 19:04:42 on dist-test-slave-9fn1"
server_key: "ffec685851b9b79ee755981d67c54e72"
server_key_iv: "057c16ca6b63b5e6c81e910dcc6c40e2"
server_key_version: "encryptionkey@0"
I20260924 19:04:42.854091 17868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:42.869169 17868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:42.873024 17868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:42.873328 17868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:42.874341 17868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:04:42.875638 17868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:04:42.875741 17868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:42.875885 17868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:04:42.875962 17868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:42.909974 17868 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34427
I20260924 19:04:42.910010 17995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34427 every 8 connection(s)
I20260924 19:04:42.911252 17868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:04:42.918009 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 17868
I20260924 19:04:42.918208 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-2/wal/instance
I20260924 19:04:42.918727 388 external_mini_cluster.cc:1428] Setting key d5c642727b939db4cd7fb2374def6458
Sep 24 19:04:42 dist-test-slave-9fn1 krb5kdc[17495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:04:42.932971 17731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:42.913792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44259 (local address 127.0.97.62:40047)
0924 19:04:42.913975 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:42.913980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:42.915156 (+ 1176us) server_negotiation.cc:407] Connection header received
0924 19:04:42.917020 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:42.917026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:42.917128 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:42.917274 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:42.919346 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:42.920262 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:42.922338 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:42.922960 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:42.923086 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:42.927351 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:42.927356 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:42.927359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:42.927390 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:42.930048 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:42.930856 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:42.930859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:42.930862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:42.930974 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:42.931411 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:42.931413 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:42.931415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:42.931653 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:42.931758 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:42.932512 (+ 754us) server_negotiation.cc:299] Negotiation successful
0924 19:04:42.932704 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 19:04:42.934156 17998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:42.914294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40047 (local address 127.0.97.3:44259)
0924 19:04:42.914997 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:42.915034 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:42.916746 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:42.917546 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:42.917560 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:42.918346 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:04:42.919142 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:42.919169 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:42.920424 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:42.920430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:42.922156 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:42.922171 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:42.923176 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:42.923183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:42.923490 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:42.924224 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:04:42.924257 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:04:42.927125 (+ 2868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:04:42.930180 (+ 3055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:04:42.930188 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:04:42.930202 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:04:42.930733 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:04:42.931077 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:04:42.931083 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:04:42.931086 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:04:42.931301 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:04:42.931766 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:04:42.931771 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:04:42.932097 (+ 326us) client_negotiation.cc:769] Sending connection context
0924 19:04:42.932375 (+ 278us) client_negotiation.cc:240] Negotiation successful
0924 19:04:42.932872 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":270,"threads_started":1}
I20260924 19:04:42.935540 17996 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40047
I20260924 19:04:42.935829 17996 heartbeater.cc:464] Registering TS with master...
I20260924 19:04:42.936527 17996 heartbeater.cc:511] Master 127.0.97.62:40047 requested a full tablet report, sending...
I20260924 19:04:42.937958 17540 ts_manager.cc:194] Registered new tserver with Master: bca3648be64f4e01b9ec93727c0d5601 (127.0.97.3:34427)
I20260924 19:04:42.938757 17540 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44259
I20260924 19:04:42.946412 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:04:42.958763 17729 heartbeater.cc:503] Master 127.0.97.62:40047 was elected leader, sending a full tablet report...
I20260924 19:04:42.963358 17731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:42.950754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45138 (local address 127.0.97.62:40047)
0924 19:04:42.950936 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:42.950945 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:42.951324 (+ 379us) server_negotiation.cc:407] Connection header received
0924 19:04:42.951448 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:42.951452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:42.951518 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:42.951602 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:42.952941 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:42.953737 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:42.955034 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:42.955475 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:42.955575 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:42.957918 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:42.957923 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:42.957926 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:42.957955 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:42.960672 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:42.961334 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:42.961346 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:42.961348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:42.961469 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:42.961866 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:42.961872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:42.961876 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:42.962276 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:42.962516 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:42.962874 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 19:04:42.963026 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 19:04:42.967111 17540 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45138:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:04:42.973958 17540 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:04:42.999164 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:42.985291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34427 (local address 127.0.0.1:51494)
0924 19:04:42.987445 (+ 2154us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:42.987497 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:42.987834 (+ 337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:42.989069 (+ 1235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:42.989079 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:42.989126 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:42.989656 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:42.989673 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:42.992310 (+ 2637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:42.992318 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:42.995527 (+ 3209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:42.995549 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:42.998052 (+ 2503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:42.998059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:42.998269 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:42.998390 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 19:04:42.998511 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 19:04:42.998711 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1786,"mutex_wait_us":29,"thread_start_us":1610,"threads_started":1}
I20260924 19:04:42.999174 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:42.985575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51494 (local address 127.0.97.3:34427)
0924 19:04:42.986681 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:42.986693 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:42.987636 (+ 943us) server_negotiation.cc:407] Connection header received
0924 19:04:42.988119 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:42.988132 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:42.988438 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:42.988631 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:42.989911 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:42.992077 (+ 2166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:42.995766 (+ 3689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:42.997517 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:42.997646 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:42.997897 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:42.998612 (+ 715us) server_negotiation.cc:299] Negotiation successful
0924 19:04:42.998796 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":879,"thread_start_us":1054,"threads_started":1}
I20260924 19:04:43.003943 17931 tablet_service.cc:1467] Processing CreateTablet for tablet 74573e11d8f84bd98e612ed2e92c6be0 (DEFAULT_TABLE table=test-table [id=b25b4810c434497392a564a1a84423a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:04:43.005365 17931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74573e11d8f84bd98e612ed2e92c6be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:04:43.006532 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:42.986416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34132 (local address 127.0.97.1:46027)
0924 19:04:42.996418 (+ 10002us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:42.996430 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:42.997374 (+ 944us) server_negotiation.cc:407] Connection header received
0924 19:04:42.997493 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:42.997520 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:42.997819 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:42.998031 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:42.998953 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:43.000323 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:43.003926 (+ 3603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:43.005398 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:43.005544 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:43.005781 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:43.006060 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 19:04:43.006213 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9774,"thread_start_us":9983,"threads_started":1}
I20260924 19:04:43.006556 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:42.985958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46027 (local address 127.0.0.1:34132)
0924 19:04:42.997072 (+ 11114us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:42.997095 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:42.997234 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:42.998151 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:42.998156 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:42.998179 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:42.998738 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:42.998750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:43.001448 (+ 2698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:43.001457 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:43.003681 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:43.003696 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:43.005678 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:43.005687 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:43.005860 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:43.005898 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 19:04:43.006005 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 19:04:43.006125 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10970,"spinlock_wait_cycles":3840,"thread_start_us":10813,"threads_started":1}
I20260924 19:04:43.007114 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:42.988470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46548 (local address 127.0.97.2:40157)
0924 19:04:42.997785 (+ 9315us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:42.997793 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:42.998588 (+ 795us) server_negotiation.cc:407] Connection header received
0924 19:04:42.998684 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:42.998690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:42.998912 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:42.999088 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:43.000433 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:43.001626 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:43.004101 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:43.005549 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:43.005661 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:43.005820 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:43.006675 (+ 855us) server_negotiation.cc:299] Negotiation successful
0924 19:04:43.006825 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9166,"thread_start_us":9030,"threads_started":1}
I20260924 19:04:43.007515 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:42.987915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40157 (local address 127.0.0.1:46548)
0924 19:04:42.998267 (+ 10352us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:42.998286 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:42.998468 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:42.999830 (+ 1362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:42.999833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:42.999854 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:43.000215 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:43.000235 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:43.001770 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:43.001777 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:43.003910 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:43.003920 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:43.005950 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:43.005955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:43.006135 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:43.006169 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 19:04:43.006263 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 19:04:43.006410 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10193,"spinlock_wait_cycles":8064,"thread_start_us":10012,"threads_started":1}
I20260924 19:04:43.010478 17798 tablet_service.cc:1467] Processing CreateTablet for tablet 74573e11d8f84bd98e612ed2e92c6be0 (DEFAULT_TABLE table=test-table [id=b25b4810c434497392a564a1a84423a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:04:43.010566 17664 tablet_service.cc:1467] Processing CreateTablet for tablet 74573e11d8f84bd98e612ed2e92c6be0 (DEFAULT_TABLE table=test-table [id=b25b4810c434497392a564a1a84423a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:04:43.011597 17798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74573e11d8f84bd98e612ed2e92c6be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:04:43.011931 17664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74573e11d8f84bd98e612ed2e92c6be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:04:43.020128 18014 tablet_bootstrap.cc:492] T 74573e11d8f84bd98e612ed2e92c6be0 P bca3648be64f4e01b9ec93727c0d5601: Bootstrap starting.
I20260924 19:04:43.023684 18014 tablet_bootstrap.cc:654] T 74573e11d8f84bd98e612ed2e92c6be0 P bca3648be64f4e01b9ec93727c0d5601: Neither blocks nor log segments found. Creating new log.
I20260924 19:04:43.023792 18015 tablet_bootstrap.cc:492] T 74573e11d8f84bd98e612ed2e92c6be0 P b711b03d0dec4a51b717e6f99a1356bc: Bootstrap starting.
I20260924 19:04:43.024919 18014 log.cc:824] T 74573e11d8f84bd98e612ed2e92c6be0 P bca3648be64f4e01b9ec93727c0d5601: Log is configured to *not* fsync() on all Append() calls
I20260924 19:04:43.028240 18015 tablet_bootstrap.cc:654] T 74573e11d8f84bd98e612ed2e92c6be0 P b711b03d0dec4a51b717e6f99a1356bc: Neither blocks nor log segments found. Creating new log.
I20260924 19:04:43.029821 18015 log.cc:824] T 74573e11d8f84bd98e612ed2e92c6be0 P b711b03d0dec4a51b717e6f99a1356bc: Log is configured to *not* fsync() on all Append() calls
I20260924 19:04:43.030372 18016 tablet_bootstrap.cc:492] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7: Bootstrap starting.
I20260924 19:04:43.033306 18014 tablet_bootstrap.cc:492] T 74573e11d8f84bd98e612ed2e92c6be0 P bca3648be64f4e01b9ec93727c0d5601: No bootstrap required, opened a new log
I20260924 19:04:43.033610 18015 tablet_bootstrap.cc:492] T 74573e11d8f84bd98e612ed2e92c6be0 P b711b03d0dec4a51b717e6f99a1356bc: No bootstrap required, opened a new log
I20260924 19:04:43.033622 18014 ts_tablet_manager.cc:1397] T 74573e11d8f84bd98e612ed2e92c6be0 P bca3648be64f4e01b9ec93727c0d5601: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260924 19:04:43.033866 18015 ts_tablet_manager.cc:1397] T 74573e11d8f84bd98e612ed2e92c6be0 P b711b03d0dec4a51b717e6f99a1356bc: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260924 19:04:43.034075 18016 tablet_bootstrap.cc:654] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7: Neither blocks nor log segments found. Creating new log.
I20260924 19:04:43.035306 18016 log.cc:824] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7: Log is configured to *not* fsync() on all Append() calls
I20260924 19:04:43.038328 18016 tablet_bootstrap.cc:492] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7: No bootstrap required, opened a new log
I20260924 19:04:43.038569 18016 ts_tablet_manager.cc:1397] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20260924 19:04:43.041407 18014 raft_consensus.cc:348] T 74573e11d8f84bd98e612ed2e92c6be0 P bca3648be64f4e01b9ec93727c0d5601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca3648be64f4e01b9ec93727c0d5601" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34427 } } peers { permanent_uuid: "b711b03d0dec4a51b717e6f99a1356bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40157 } } peers { permanent_uuid: "50d35e55a23943a59817d2794dce4ff7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46027 } }
I20260924 19:04:43.041818 18014 raft_consensus.cc:374] T 74573e11d8f84bd98e612ed2e92c6be0 P bca3648be64f4e01b9ec93727c0d5601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:04:43.041962 18014 raft_consensus.cc:729] T 74573e11d8f84bd98e612ed2e92c6be0 P bca3648be64f4e01b9ec93727c0d5601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bca3648be64f4e01b9ec93727c0d5601, State: Initialized, Role: FOLLOWER
I20260924 19:04:43.042567 18014 consensus_queue.cc:260] T 74573e11d8f84bd98e612ed2e92c6be0 P bca3648be64f4e01b9ec93727c0d5601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca3648be64f4e01b9ec93727c0d5601" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34427 } } peers { permanent_uuid: "b711b03d0dec4a51b717e6f99a1356bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40157 } } peers { permanent_uuid: "50d35e55a23943a59817d2794dce4ff7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46027 } }
I20260924 19:04:43.043596 17996 heartbeater.cc:503] Master 127.0.97.62:40047 was elected leader, sending a full tablet report...
I20260924 19:04:43.044234 18014 ts_tablet_manager.cc:1428] T 74573e11d8f84bd98e612ed2e92c6be0 P bca3648be64f4e01b9ec93727c0d5601: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260924 19:04:43.044021 18015 raft_consensus.cc:348] T 74573e11d8f84bd98e612ed2e92c6be0 P b711b03d0dec4a51b717e6f99a1356bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca3648be64f4e01b9ec93727c0d5601" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34427 } } peers { permanent_uuid: "b711b03d0dec4a51b717e6f99a1356bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40157 } } peers { permanent_uuid: "50d35e55a23943a59817d2794dce4ff7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46027 } }
I20260924 19:04:43.044402 18015 raft_consensus.cc:374] T 74573e11d8f84bd98e612ed2e92c6be0 P b711b03d0dec4a51b717e6f99a1356bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:04:43.044480 18015 raft_consensus.cc:729] T 74573e11d8f84bd98e612ed2e92c6be0 P b711b03d0dec4a51b717e6f99a1356bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b711b03d0dec4a51b717e6f99a1356bc, State: Initialized, Role: FOLLOWER
I20260924 19:04:43.045178 18015 consensus_queue.cc:260] T 74573e11d8f84bd98e612ed2e92c6be0 P b711b03d0dec4a51b717e6f99a1356bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca3648be64f4e01b9ec93727c0d5601" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34427 } } peers { permanent_uuid: "b711b03d0dec4a51b717e6f99a1356bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40157 } } peers { permanent_uuid: "50d35e55a23943a59817d2794dce4ff7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46027 } }
I20260924 19:04:43.048470 17863 heartbeater.cc:503] Master 127.0.97.62:40047 was elected leader, sending a full tablet report...
I20260924 19:04:43.048611 18016 raft_consensus.cc:348] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca3648be64f4e01b9ec93727c0d5601" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34427 } } peers { permanent_uuid: "b711b03d0dec4a51b717e6f99a1356bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40157 } } peers { permanent_uuid: "50d35e55a23943a59817d2794dce4ff7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46027 } }
I20260924 19:04:43.049018 18016 raft_consensus.cc:374] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:04:43.049093 18015 ts_tablet_manager.cc:1428] T 74573e11d8f84bd98e612ed2e92c6be0 P b711b03d0dec4a51b717e6f99a1356bc: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260924 19:04:43.049108 18016 raft_consensus.cc:729] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50d35e55a23943a59817d2794dce4ff7, State: Initialized, Role: FOLLOWER
I20260924 19:04:43.049919 18016 consensus_queue.cc:260] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca3648be64f4e01b9ec93727c0d5601" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34427 } } peers { permanent_uuid: "b711b03d0dec4a51b717e6f99a1356bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40157 } } peers { permanent_uuid: "50d35e55a23943a59817d2794dce4ff7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46027 } }
I20260924 19:04:43.051678 18016 ts_tablet_manager.cc:1428] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7: Time spent starting tablet: real 0.013s user 0.004s sys 0.009s
I20260924 19:04:43.112483 18022 raft_consensus.cc:482] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:04:43.112816 18022 raft_consensus.cc:504] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca3648be64f4e01b9ec93727c0d5601" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34427 } } peers { permanent_uuid: "b711b03d0dec4a51b717e6f99a1356bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40157 } } peers { permanent_uuid: "50d35e55a23943a59817d2794dce4ff7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46027 } }
I20260924 19:04:43.114548 18022 leader_election.cc:288] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bca3648be64f4e01b9ec93727c0d5601 (127.0.97.3:34427), b711b03d0dec4a51b717e6f99a1356bc (127.0.97.2:40157)
I20260924 19:04:43.121439 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:43.114915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41703 (local address 127.0.97.3:34427)
0924 19:04:43.115163 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:43.115170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:43.115561 (+ 391us) server_negotiation.cc:407] Connection header received
0924 19:04:43.115777 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:43.115787 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:43.115894 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:43.115992 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:43.116674 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:43.117686 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:43.119647 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:43.120489 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:43.120584 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:43.120653 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:43.121044 (+ 391us) server_negotiation.cc:299] Negotiation successful
0924 19:04:43.121176 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 19:04:43.121487 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:43.114850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34427 (local address 127.0.97.1:41703)
0924 19:04:43.115415 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:43.115431 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:43.115608 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:43.116058 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:43.116063 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:43.116090 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:43.116506 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:43.116515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:43.117837 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:43.117843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:43.119478 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:43.119490 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:43.120687 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:43.120693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:43.120868 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:43.120903 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 19:04:43.120995 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 19:04:43.121085 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":52,"thread_start_us":285,"threads_started":1}
I20260924 19:04:43.121652 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:43.115714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49267 (local address 127.0.97.2:40157)
0924 19:04:43.115952 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:43.115960 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:43.116278 (+ 318us) server_negotiation.cc:407] Connection header received
0924 19:04:43.116429 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:43.116433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:43.116489 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:43.116559 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:43.117117 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:43.118103 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:43.119877 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:43.120686 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:43.120818 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:43.120917 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:43.121243 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 19:04:43.121362 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 19:04:43.121960 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:43.115564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40157 (local address 127.0.97.1:49267)
0924 19:04:43.116158 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:43.116176 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:43.116306 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:43.116600 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:43.116608 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:43.116634 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:43.116965 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:43.116975 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:43.118243 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:43.118246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:43.119722 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:43.119737 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:43.120901 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:43.120906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:43.121073 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:43.121104 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 19:04:43.121194 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 19:04:43.121316 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"mutex_wait_us":2,"thread_start_us":273,"threads_started":1}
I20260924 19:04:43.122651 17951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74573e11d8f84bd98e612ed2e92c6be0" candidate_uuid: "50d35e55a23943a59817d2794dce4ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bca3648be64f4e01b9ec93727c0d5601" is_pre_election: true
I20260924 19:04:43.122877 17818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74573e11d8f84bd98e612ed2e92c6be0" candidate_uuid: "50d35e55a23943a59817d2794dce4ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b711b03d0dec4a51b717e6f99a1356bc" is_pre_election: true
I20260924 19:04:43.123221 17951 raft_consensus.cc:2454] T 74573e11d8f84bd98e612ed2e92c6be0 P bca3648be64f4e01b9ec93727c0d5601 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50d35e55a23943a59817d2794dce4ff7 in term 0.
I20260924 19:04:43.123351 17818 raft_consensus.cc:2454] T 74573e11d8f84bd98e612ed2e92c6be0 P b711b03d0dec4a51b717e6f99a1356bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50d35e55a23943a59817d2794dce4ff7 in term 0.
I20260924 19:04:43.123879 17617 leader_election.cc:302] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50d35e55a23943a59817d2794dce4ff7, bca3648be64f4e01b9ec93727c0d5601; no voters:
I20260924 19:04:43.124164 18022 raft_consensus.cc:2781] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:04:43.124280 18022 raft_consensus.cc:482] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:04:43.124439 18022 raft_consensus.cc:3037] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:04:43.126667 18022 raft_consensus.cc:504] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca3648be64f4e01b9ec93727c0d5601" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34427 } } peers { permanent_uuid: "b711b03d0dec4a51b717e6f99a1356bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40157 } } peers { permanent_uuid: "50d35e55a23943a59817d2794dce4ff7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46027 } }
I20260924 19:04:43.127219 18022 leader_election.cc:288] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7 [CANDIDATE]: Term 1 election: Requested vote from peers bca3648be64f4e01b9ec93727c0d5601 (127.0.97.3:34427), b711b03d0dec4a51b717e6f99a1356bc (127.0.97.2:40157)
I20260924 19:04:43.127629 17951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74573e11d8f84bd98e612ed2e92c6be0" candidate_uuid: "50d35e55a23943a59817d2794dce4ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bca3648be64f4e01b9ec93727c0d5601"
I20260924 19:04:43.127676 17818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74573e11d8f84bd98e612ed2e92c6be0" candidate_uuid: "50d35e55a23943a59817d2794dce4ff7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b711b03d0dec4a51b717e6f99a1356bc"
I20260924 19:04:43.127807 17951 raft_consensus.cc:3037] T 74573e11d8f84bd98e612ed2e92c6be0 P bca3648be64f4e01b9ec93727c0d5601 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:04:43.127898 17818 raft_consensus.cc:3037] T 74573e11d8f84bd98e612ed2e92c6be0 P b711b03d0dec4a51b717e6f99a1356bc [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:04:43.130067 17951 raft_consensus.cc:2454] T 74573e11d8f84bd98e612ed2e92c6be0 P bca3648be64f4e01b9ec93727c0d5601 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50d35e55a23943a59817d2794dce4ff7 in term 1.
I20260924 19:04:43.130132 17818 raft_consensus.cc:2454] T 74573e11d8f84bd98e612ed2e92c6be0 P b711b03d0dec4a51b717e6f99a1356bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50d35e55a23943a59817d2794dce4ff7 in term 1.
I20260924 19:04:43.130445 17617 leader_election.cc:302] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50d35e55a23943a59817d2794dce4ff7, bca3648be64f4e01b9ec93727c0d5601; no voters:
I20260924 19:04:43.130757 18022 raft_consensus.cc:2781] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:04:43.131167 18022 raft_consensus.cc:686] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7 [term 1 LEADER]: Becoming Leader. State: Replica: 50d35e55a23943a59817d2794dce4ff7, State: Running, Role: LEADER
I20260924 19:04:43.131659 18022 consensus_queue.cc:237] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca3648be64f4e01b9ec93727c0d5601" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34427 } } peers { permanent_uuid: "b711b03d0dec4a51b717e6f99a1356bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40157 } } peers { permanent_uuid: "50d35e55a23943a59817d2794dce4ff7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46027 } }
I20260924 19:04:43.136058 17539 catalog_manager.cc:5495] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50d35e55a23943a59817d2794dce4ff7 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "50d35e55a23943a59817d2794dce4ff7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca3648be64f4e01b9ec93727c0d5601" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b711b03d0dec4a51b717e6f99a1356bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50d35e55a23943a59817d2794dce4ff7" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46027 } health_report { overall_health: HEALTHY } } }
I20260924 19:04:43.160287 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:43.154681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34138 (local address 127.0.97.1:46027)
0924 19:04:43.154900 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:43.154905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:43.154923 (+ 18us) server_negotiation.cc:407] Connection header received
0924 19:04:43.155069 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:43.155073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:43.155148 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:43.155272 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:04:43.155564 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:43.156252 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:43.158032 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:43.158467 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:43.158560 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:43.159138 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:04:43.159642 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:04:43.159759 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:43.159906 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 19:04:43.160005 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260924 19:04:43.162787 17997 tablet.cc:2369] T 74573e11d8f84bd98e612ed2e92c6be0 P bca3648be64f4e01b9ec93727c0d5601: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:04:43.163892 17864 tablet.cc:2369] T 74573e11d8f84bd98e612ed2e92c6be0 P b711b03d0dec4a51b717e6f99a1356bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:04:43.168582 17951 raft_consensus.cc:1261] T 74573e11d8f84bd98e612ed2e92c6be0 P bca3648be64f4e01b9ec93727c0d5601 [term 1 FOLLOWER]: Refusing update from remote peer 50d35e55a23943a59817d2794dce4ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:04:43.168630 17818 raft_consensus.cc:1261] T 74573e11d8f84bd98e612ed2e92c6be0 P b711b03d0dec4a51b717e6f99a1356bc [term 1 FOLLOWER]: Refusing update from remote peer 50d35e55a23943a59817d2794dce4ff7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:04:43.169551 18025 consensus_queue.cc:1035] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bca3648be64f4e01b9ec93727c0d5601" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:04:43.169941 18022 consensus_queue.cc:1035] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b711b03d0dec4a51b717e6f99a1356bc" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:04:43.178895 17730 tablet.cc:2369] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:04:43.188561 18030 mvcc.cc:205] Tried to move back new op lower bound from 7332973294248169472 to 7332973294110199808. Current Snapshot: MvccSnapshot[applied={T|T < 7332973294248169472}]
I20260924 19:04:43.204115 17539 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:04:43.204418 17539 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45138:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:04:43.207732 17539 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 24fb55e0a9da4f1bb881d825d851c962: Sending DeleteTablet for 3 replicas of tablet 74573e11d8f84bd98e612ed2e92c6be0
I20260924 19:04:43.208969 17664 tablet_service.cc:1514] Processing DeleteTablet for tablet 74573e11d8f84bd98e612ed2e92c6be0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:04:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34132
I20260924 19:04:43.209218 17798 tablet_service.cc:1514] Processing DeleteTablet for tablet 74573e11d8f84bd98e612ed2e92c6be0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:04:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46548
I20260924 19:04:43.209227 17931 tablet_service.cc:1514] Processing DeleteTablet for tablet 74573e11d8f84bd98e612ed2e92c6be0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:04:43 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51494
I20260924 19:04:43.211153 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 17601
I20260924 19:04:43.218045 18040 tablet_replica.cc:330] T 74573e11d8f84bd98e612ed2e92c6be0 P bca3648be64f4e01b9ec93727c0d5601: stopping tablet replica
I20260924 19:04:43.218590 18040 raft_consensus.cc:2229] T 74573e11d8f84bd98e612ed2e92c6be0 P bca3648be64f4e01b9ec93727c0d5601 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:04:43.219192 18040 raft_consensus.cc:2258] T 74573e11d8f84bd98e612ed2e92c6be0 P bca3648be64f4e01b9ec93727c0d5601 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:04:43.222491 18040 ts_tablet_manager.cc:1905] T 74573e11d8f84bd98e612ed2e92c6be0 P bca3648be64f4e01b9ec93727c0d5601: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:04:43.224308 18039 tablet_replica.cc:330] T 74573e11d8f84bd98e612ed2e92c6be0 P b711b03d0dec4a51b717e6f99a1356bc: stopping tablet replica
I20260924 19:04:43.224699 18039 raft_consensus.cc:2229] T 74573e11d8f84bd98e612ed2e92c6be0 P b711b03d0dec4a51b717e6f99a1356bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:04:43.225119 18039 raft_consensus.cc:2258] T 74573e11d8f84bd98e612ed2e92c6be0 P b711b03d0dec4a51b717e6f99a1356bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:04:43.228555 18039 ts_tablet_manager.cc:1905] T 74573e11d8f84bd98e612ed2e92c6be0 P b711b03d0dec4a51b717e6f99a1356bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:04:43.234272 18039 ts_tablet_manager.cc:1918] T 74573e11d8f84bd98e612ed2e92c6be0 P b711b03d0dec4a51b717e6f99a1356bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:04:43.234570 18039 log.cc:1196] T 74573e11d8f84bd98e612ed2e92c6be0 P b711b03d0dec4a51b717e6f99a1356bc: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-1/wal/wals/74573e11d8f84bd98e612ed2e92c6be0
I20260924 19:04:43.235314 18039 ts_tablet_manager.cc:1939] T 74573e11d8f84bd98e612ed2e92c6be0 P b711b03d0dec4a51b717e6f99a1356bc: Deleting consensus metadata
I20260924 19:04:43.238343 17524 catalog_manager.cc:4841] TS b711b03d0dec4a51b717e6f99a1356bc (127.0.97.2:40157): tablet 74573e11d8f84bd98e612ed2e92c6be0 (table test-table [id=b25b4810c434497392a564a1a84423a6]) successfully deleted
I20260924 19:04:43.239531 18040 ts_tablet_manager.cc:1918] T 74573e11d8f84bd98e612ed2e92c6be0 P bca3648be64f4e01b9ec93727c0d5601: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:04:43.239697 18040 log.cc:1196] T 74573e11d8f84bd98e612ed2e92c6be0 P bca3648be64f4e01b9ec93727c0d5601: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-2/wal/wals/74573e11d8f84bd98e612ed2e92c6be0
I20260924 19:04:43.240327 18040 ts_tablet_manager.cc:1939] T 74573e11d8f84bd98e612ed2e92c6be0 P bca3648be64f4e01b9ec93727c0d5601: Deleting consensus metadata
I20260924 19:04:43.242090 17524 catalog_manager.cc:4841] TS bca3648be64f4e01b9ec93727c0d5601 (127.0.97.3:34427): tablet 74573e11d8f84bd98e612ed2e92c6be0 (table test-table [id=b25b4810c434497392a564a1a84423a6]) successfully deleted
I20260924 19:04:43.242364 18038 tablet_replica.cc:330] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7: stopping tablet replica
I20260924 19:04:43.242835 18038 raft_consensus.cc:2229] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:04:43.243511 18038 raft_consensus.cc:2258] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:04:43.245414 18038 ts_tablet_manager.cc:1905] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 19:04:43 dist-test-slave-9fn1 krb5kdc[17495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 19:04:43.252992 18038 ts_tablet_manager.cc:1918] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:04:43.253311 18038 log.cc:1196] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276386554690-388-0/minicluster-data/ts-0/wal/wals/74573e11d8f84bd98e612ed2e92c6be0
I20260924 19:04:43.253827 18011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:43.212508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34154 (local address 127.0.97.1:46027)
0924 19:04:43.212687 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:43.212694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:43.232086 (+ 19392us) server_negotiation.cc:407] Connection header received
0924 19:04:43.232553 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:43.232560 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:43.232651 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:43.232740 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:43.234560 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:43.235850 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:43.242199 (+ 6349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:43.242803 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:43.242881 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:43.247586 (+ 4705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:43.247599 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:43.247612 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:43.247650 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:43.250938 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:43.251631 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:43.251638 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:43.251640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:43.251797 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:43.252194 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:43.252198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:43.252199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:43.252526 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:43.252817 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:43.253137 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 19:04:43.253393 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 19:04:43.253929 18038 ts_tablet_manager.cc:1939] T 74573e11d8f84bd98e612ed2e92c6be0 P 50d35e55a23943a59817d2794dce4ff7: Deleting consensus metadata
I20260924 19:04:43.254570 17724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:04:43.354477 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 17601
I20260924 19:04:43.354624 17526 catalog_manager.cc:4841] TS 50d35e55a23943a59817d2794dce4ff7 (127.0.97.1:46027): tablet 74573e11d8f84bd98e612ed2e92c6be0 (table test-table [id=b25b4810c434497392a564a1a84423a6]) successfully deleted
I20260924 19:04:43.373363 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 17735
Sep 24 19:04:43 dist-test-slave-9fn1 krb5kdc[17495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 19:04:43.387874 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:43.374148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46564 (local address 127.0.97.2:40157)
0924 19:04:43.374316 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:43.374320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:43.374337 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:04:43.374502 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:43.374506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:43.374570 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:43.374658 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:43.375842 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:43.376554 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:43.377950 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:43.378330 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:43.378421 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:43.382342 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:43.382354 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:43.382369 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:43.382400 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:43.385260 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:43.386056 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:43.386063 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:43.386065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:43.386197 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:43.386570 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:43.386573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:43.386574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:43.386944 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:43.387209 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:43.387364 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 19:04:43.387525 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 19:04:43.388491 17858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:04:43.475402 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 17735
I20260924 19:04:43.492265 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 17868
Sep 24 19:04:43 dist-test-slave-9fn1 krb5kdc[17495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 19:04:43.507062 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:43.493018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51496 (local address 127.0.97.3:34427)
0924 19:04:43.493214 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:43.493221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:43.493241 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:04:43.493435 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:43.493447 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:43.493564 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:43.493676 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:43.494996 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:43.495754 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:43.497053 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:43.497452 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:43.497644 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:43.502131 (+ 4487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:43.502153 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:43.502168 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:43.502201 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:43.504716 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:43.505302 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:43.505309 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:43.505311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:43.505445 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:43.505794 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:43.505797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:43.505799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:43.506135 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:43.506362 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:43.506594 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 19:04:43.506764 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 19:04:43.507752 17991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:04:43.596148 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 17868
I20260924 19:04:43.612885 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 17508
I20260924 19:04:43.614060 17570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:04:43.720920 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 17508
2026-09-24T19:04:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4162 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T19:04:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:04:43Z Disabled control of system clock
I20260924 19:04:43.794618 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34955
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:04:44.115353 18049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:44.115628 18049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:44.115708 18049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:44.122727 18049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:04:44.122843 18049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:44.122881 18049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:44.122931 18049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:04:44.122974 18049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:04:44.131744 18049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34955
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:44.132921 18049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:44.134688 18049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:44.146349 18057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:44.146418 18055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:44.146545 18054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:44.147585 18049 server_base.cc:1034] running on GCE node
I20260924 19:04:44.148366 18049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:44.149817 18049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:44.151057 18049 hybrid_clock.cc:648] HybridClock initialized: now 1790276684150992 us; error 57 us; skew 500 ppm
I20260924 19:04:44.151403 18049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:04:44.159578 18049 webserver.cc:460] Webserver started at http://127.0.97.62:45481/ using document root <none> and password file <none>
I20260924 19:04:44.160271 18049 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:44.160406 18049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:44.160728 18049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:04:44.163612 18049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "2a465d0020f94af5964a0849126a0286"
format_stamp: "Formatted at 2026-09-24 19:04:44 on dist-test-slave-9fn1"
server_key: "76ec9abb026f81428087e1e9ba2fd30c"
server_key_iv: "63687fdf45a66571365f47deb89bafce"
server_key_version: "encryptionkey@0"
I20260924 19:04:44.164397 18049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "2a465d0020f94af5964a0849126a0286"
format_stamp: "Formatted at 2026-09-24 19:04:44 on dist-test-slave-9fn1"
server_key: "76ec9abb026f81428087e1e9ba2fd30c"
server_key_iv: "63687fdf45a66571365f47deb89bafce"
server_key_version: "encryptionkey@0"
I20260924 19:04:44.169322 18049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 19:04:44.172773 18063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:44.174317 18049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:04:44.174506 18049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "2a465d0020f94af5964a0849126a0286"
format_stamp: "Formatted at 2026-09-24 19:04:44 on dist-test-slave-9fn1"
server_key: "76ec9abb026f81428087e1e9ba2fd30c"
server_key_iv: "63687fdf45a66571365f47deb89bafce"
server_key_version: "encryptionkey@0"
I20260924 19:04:44.174718 18049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:44.191795 18049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:44.192584 18049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:44.192858 18049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:44.210750 18049 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34955
I20260924 19:04:44.210780 18114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34955 every 8 connection(s)
I20260924 19:04:44.212033 18049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:04:44.214977 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 18049
I20260924 19:04:44.215147 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:04:44.215692 388 external_mini_cluster.cc:1428] Setting key 5cc6b0912845ab68aaadcbc39005f926
I20260924 19:04:44.216025 18115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:04:44.229406 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:44.217558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37702 (local address 127.0.97.62:34955)
0924 19:04:44.218511 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:44.218521 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:44.218563 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:04:44.220532 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:44.220556 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:44.221004 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:44.221318 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:44.221923 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:44.223188 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:44.225167 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:44.225918 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:44.226070 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:44.227253 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:44.227260 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:44.227275 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:04:44.227796 (+ 521us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:44.227876 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:04:44.227896 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:44.227979 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:44.228203 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 19:04:44.228525 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":353,"threads_started":1}
I20260924 19:04:44.229704 18115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a465d0020f94af5964a0849126a0286: Bootstrap starting.
I20260924 19:04:44.233115 18115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a465d0020f94af5964a0849126a0286: Neither blocks nor log segments found. Creating new log.
I20260924 19:04:44.234310 18115 log.cc:824] T 00000000000000000000000000000000 P 2a465d0020f94af5964a0849126a0286: Log is configured to *not* fsync() on all Append() calls
I20260924 19:04:44.236874 18115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a465d0020f94af5964a0849126a0286: No bootstrap required, opened a new log
I20260924 19:04:44.244306 18115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a465d0020f94af5964a0849126a0286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a465d0020f94af5964a0849126a0286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34955 } }
I20260924 19:04:44.244688 18115 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a465d0020f94af5964a0849126a0286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:04:44.244813 18115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a465d0020f94af5964a0849126a0286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a465d0020f94af5964a0849126a0286, State: Initialized, Role: FOLLOWER
I20260924 19:04:44.245388 18115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a465d0020f94af5964a0849126a0286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a465d0020f94af5964a0849126a0286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34955 } }
I20260924 19:04:44.245662 18115 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a465d0020f94af5964a0849126a0286 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:04:44.245795 18115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a465d0020f94af5964a0849126a0286 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:04:44.245966 18115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a465d0020f94af5964a0849126a0286 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:04:44.248359 18115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a465d0020f94af5964a0849126a0286 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a465d0020f94af5964a0849126a0286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34955 } }
I20260924 19:04:44.248862 18115 leader_election.cc:302] T 00000000000000000000000000000000 P 2a465d0020f94af5964a0849126a0286 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a465d0020f94af5964a0849126a0286; no voters:
I20260924 19:04:44.249432 18115 leader_election.cc:288] T 00000000000000000000000000000000 P 2a465d0020f94af5964a0849126a0286 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:04:44.249624 18120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a465d0020f94af5964a0849126a0286 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:04:44.250350 18120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a465d0020f94af5964a0849126a0286 [term 1 LEADER]: Becoming Leader. State: Replica: 2a465d0020f94af5964a0849126a0286, State: Running, Role: LEADER
I20260924 19:04:44.250828 18120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a465d0020f94af5964a0849126a0286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a465d0020f94af5964a0849126a0286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34955 } }
I20260924 19:04:44.251559 18115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a465d0020f94af5964a0849126a0286 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:04:44.253729 18122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a465d0020f94af5964a0849126a0286 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a465d0020f94af5964a0849126a0286. Latest consensus state: current_term: 1 leader_uuid: "2a465d0020f94af5964a0849126a0286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a465d0020f94af5964a0849126a0286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34955 } } }
I20260924 19:04:44.254310 18122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a465d0020f94af5964a0849126a0286 [sys.catalog]: This master's current role is: LEADER
I20260924 19:04:44.254114 18121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a465d0020f94af5964a0849126a0286 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a465d0020f94af5964a0849126a0286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a465d0020f94af5964a0849126a0286" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34955 } } }
I20260924 19:04:44.257902 18121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a465d0020f94af5964a0849126a0286 [sys.catalog]: This master's current role is: LEADER
I20260924 19:04:44.260710 18128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:04:44.266340 18128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:04:44.275604 18128 catalog_manager.cc:1347] Generated new cluster ID: 3565647763164e85b0974b825532f909
I20260924 19:04:44.275733 18128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:04:44.293282 18128 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:04:44.295142 18128 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:04:44.306051 18128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a465d0020f94af5964a0849126a0286: Generated new TSK 0
I20260924 19:04:44.307164 18128 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:04:44.318813 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34955
--builtin_ntp_servers=127.0.97.20:35261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:04:44.649768 18139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:44.650019 18139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:44.650205 18139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:44.657333 18139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:44.657423 18139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:44.657644 18139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:04:44.666867 18139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:44.668072 18139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:44.669845 18139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:44.683131 18144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:44.683166 18145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:44.683188 18147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:44.683389 18139 server_base.cc:1034] running on GCE node
I20260924 19:04:44.684322 18139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:44.685259 18139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:44.686496 18139 hybrid_clock.cc:648] HybridClock initialized: now 1790276684686410 us; error 79 us; skew 500 ppm
I20260924 19:04:44.686808 18139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:04:44.688937 18139 webserver.cc:460] Webserver started at http://127.0.97.1:40329/ using document root <none> and password file <none>
I20260924 19:04:44.689685 18139 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:44.689812 18139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:44.690157 18139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:04:44.693246 18139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-0/data/instance:
uuid: "9aa402645fe24bd7813e0076a3aedb65"
format_stamp: "Formatted at 2026-09-24 19:04:44 on dist-test-slave-9fn1"
server_key: "20e4064cbfca595613ae1d9ef57f4a27"
server_key_iv: "c52d12c20100128ee38a0d00d30f7b3b"
server_key_version: "encryptionkey@0"
I20260924 19:04:44.694108 18139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9aa402645fe24bd7813e0076a3aedb65"
format_stamp: "Formatted at 2026-09-24 19:04:44 on dist-test-slave-9fn1"
server_key: "20e4064cbfca595613ae1d9ef57f4a27"
server_key_iv: "c52d12c20100128ee38a0d00d30f7b3b"
server_key_version: "encryptionkey@0"
I20260924 19:04:44.699509 18139 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:04:44.703107 18153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:44.704445 18139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:04:44.704613 18139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "9aa402645fe24bd7813e0076a3aedb65"
format_stamp: "Formatted at 2026-09-24 19:04:44 on dist-test-slave-9fn1"
server_key: "20e4064cbfca595613ae1d9ef57f4a27"
server_key_iv: "c52d12c20100128ee38a0d00d30f7b3b"
server_key_version: "encryptionkey@0"
I20260924 19:04:44.704845 18139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:44.716683 18139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:44.717414 18139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:44.717705 18139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:44.718582 18139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:04:44.719962 18139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:04:44.720088 18139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:44.720228 18139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:04:44.720314 18139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:44.751721 18139 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38141
I20260924 19:04:44.751818 18265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38141 every 8 connection(s)
I20260924 19:04:44.753005 18139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:04:44.755012 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 18139
I20260924 19:04:44.755203 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-0/wal/instance
I20260924 19:04:44.755710 388 external_mini_cluster.cc:1428] Setting key 0ace2c6695e0737c398437b4df55600d
I20260924 19:04:44.758628 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34955
--builtin_ntp_servers=127.0.97.20:35261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 19:04:44.768114 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:44.755764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40309 (local address 127.0.97.62:34955)
0924 19:04:44.756235 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:44.756241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:44.757690 (+ 1449us) server_negotiation.cc:407] Connection header received
0924 19:04:44.759696 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:44.759704 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:44.759799 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:44.759920 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:44.761315 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:44.762504 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:44.764729 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:44.765469 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:44.765634 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:44.767278 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:44.767283 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:44.767286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:04:44.767305 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:44.767350 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:04:44.767362 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:44.767459 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:44.767697 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 19:04:44.767797 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":217,"threads_started":1}
I20260924 19:04:44.769121 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:44.756418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34955 (local address 127.0.97.1:40309)
0924 19:04:44.757542 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:44.757593 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:44.759433 (+ 1840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:44.760215 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:44.760224 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:44.760340 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:04:44.761090 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:44.761109 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:44.762681 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:44.762688 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:44.764570 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:44.764587 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:44.765968 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:44.765978 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:44.766208 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:44.766947 (+ 739us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:04:44.766950 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:04:44.766990 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:04:44.767069 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:04:44.767460 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:04:44.767474 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:04:44.767483 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 19:04:44.767676 (+ 193us) client_negotiation.cc:240] Negotiation successful
0924 19:04:44.768074 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":412,"threads_started":1}
I20260924 19:04:44.770771 18266 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34955
I20260924 19:04:44.771070 18266 heartbeater.cc:464] Registering TS with master...
I20260924 19:04:44.771775 18266 heartbeater.cc:511] Master 127.0.97.62:34955 requested a full tablet report, sending...
I20260924 19:04:44.773911 18080 ts_manager.cc:194] Registered new tserver with Master: 9aa402645fe24bd7813e0076a3aedb65 (127.0.97.1:38141)
I20260924 19:04:44.775316 18080 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:40309
W20260924 19:04:45.079880 18270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:45.080137 18270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:45.080219 18270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:45.087119 18270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:45.087208 18270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:45.087320 18270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:04:45.095974 18270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:45.097039 18270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:45.098758 18270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:45.111122 18275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:45.111166 18276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:45.111353 18278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:45.112272 18270 server_base.cc:1034] running on GCE node
I20260924 19:04:45.112782 18270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:45.113668 18270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:45.114897 18270 hybrid_clock.cc:648] HybridClock initialized: now 1790276685114855 us; error 51 us; skew 500 ppm
I20260924 19:04:45.115199 18270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:04:45.117213 18270 webserver.cc:460] Webserver started at http://127.0.97.2:43767/ using document root <none> and password file <none>
I20260924 19:04:45.117937 18270 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:45.118057 18270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:45.118372 18270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:04:45.121595 18270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-1/data/instance:
uuid: "0d99dd1c7b3e4e9489e6629b464b2c3f"
format_stamp: "Formatted at 2026-09-24 19:04:45 on dist-test-slave-9fn1"
server_key: "cf825462c4b3e5cf0281c19e7c9585d2"
server_key_iv: "4ed18b388b0864401c4faa83e947f282"
server_key_version: "encryptionkey@0"
I20260924 19:04:45.122396 18270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0d99dd1c7b3e4e9489e6629b464b2c3f"
format_stamp: "Formatted at 2026-09-24 19:04:45 on dist-test-slave-9fn1"
server_key: "cf825462c4b3e5cf0281c19e7c9585d2"
server_key_iv: "4ed18b388b0864401c4faa83e947f282"
server_key_version: "encryptionkey@0"
I20260924 19:04:45.127045 18270 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260924 19:04:45.130241 18284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:45.131479 18270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:04:45.131650 18270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "0d99dd1c7b3e4e9489e6629b464b2c3f"
format_stamp: "Formatted at 2026-09-24 19:04:45 on dist-test-slave-9fn1"
server_key: "cf825462c4b3e5cf0281c19e7c9585d2"
server_key_iv: "4ed18b388b0864401c4faa83e947f282"
server_key_version: "encryptionkey@0"
I20260924 19:04:45.131852 18270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:45.152542 18270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:45.153366 18270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:45.153688 18270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:45.154587 18270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:04:45.156054 18270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:04:45.156172 18270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:45.156283 18270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:04:45.156373 18270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:45.188555 18270 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35925
I20260924 19:04:45.188652 18396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35925 every 8 connection(s)
I20260924 19:04:45.189815 18270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:04:45.193320 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 18270
I20260924 19:04:45.193544 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-1/wal/instance
I20260924 19:04:45.194056 388 external_mini_cluster.cc:1428] Setting key e5a87e48ee99cfe528abebb456bfaff8
I20260924 19:04:45.196859 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34955
--builtin_ntp_servers=127.0.97.20:35261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 19:04:45.202454 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:45.192314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53127 (local address 127.0.97.62:34955)
0924 19:04:45.192487 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:45.192491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:45.194030 (+ 1539us) server_negotiation.cc:407] Connection header received
0924 19:04:45.196073 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:45.196078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:45.196162 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:45.196275 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:45.197395 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.198146 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.199687 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.200109 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.200244 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:45.201643 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:45.201648 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:45.201651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:04:45.201669 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:45.201728 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:04:45.201746 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:45.201833 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:45.202053 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 19:04:45.202147 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 19:04:45.203425 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:45.192791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34955 (local address 127.0.97.2:53127)
0924 19:04:45.193873 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:45.193925 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:45.195816 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:45.196506 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:45.196514 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:45.196596 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:04:45.197213 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:45.197231 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.198296 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.198302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:45.199542 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:45.199553 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.200337 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.200343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:45.200599 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:45.201306 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:04:45.201308 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:04:45.201348 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:04:45.201423 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:04:45.201843 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:04:45.201855 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:04:45.201869 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 19:04:45.202044 (+ 175us) client_negotiation.cc:240] Negotiation successful
0924 19:04:45.202454 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":339,"threads_started":1}
I20260924 19:04:45.204924 18397 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34955
I20260924 19:04:45.205207 18397 heartbeater.cc:464] Registering TS with master...
I20260924 19:04:45.205965 18397 heartbeater.cc:511] Master 127.0.97.62:34955 requested a full tablet report, sending...
I20260924 19:04:45.207291 18080 ts_manager.cc:194] Registered new tserver with Master: 0d99dd1c7b3e4e9489e6629b464b2c3f (127.0.97.2:35925)
I20260924 19:04:45.208132 18080 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:53127
W20260924 19:04:45.517107 18400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:45.517421 18400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:45.517653 18400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:45.524744 18400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:45.524874 18400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:45.525061 18400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:04:45.533999 18400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:45.535336 18400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:45.537146 18400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:45.549901 18405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:45.549877 18406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:45.549849 18408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:45.550680 18400 server_base.cc:1034] running on GCE node
I20260924 19:04:45.551288 18400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:45.552260 18400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:45.553470 18400 hybrid_clock.cc:648] HybridClock initialized: now 1790276685553438 us; error 29 us; skew 500 ppm
I20260924 19:04:45.553820 18400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:04:45.555954 18400 webserver.cc:460] Webserver started at http://127.0.97.3:39237/ using document root <none> and password file <none>
I20260924 19:04:45.556595 18400 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:45.556701 18400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:45.557039 18400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:04:45.559819 18400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-2/data/instance:
uuid: "d6e3444ae531472881c8c07e637e61ac"
format_stamp: "Formatted at 2026-09-24 19:04:45 on dist-test-slave-9fn1"
server_key: "5bdbf5160634a9180529684aa4a85902"
server_key_iv: "051ef0bbad1ae1b65ce293a29f8213cf"
server_key_version: "encryptionkey@0"
I20260924 19:04:45.560573 18400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d6e3444ae531472881c8c07e637e61ac"
format_stamp: "Formatted at 2026-09-24 19:04:45 on dist-test-slave-9fn1"
server_key: "5bdbf5160634a9180529684aa4a85902"
server_key_iv: "051ef0bbad1ae1b65ce293a29f8213cf"
server_key_version: "encryptionkey@0"
I20260924 19:04:45.565667 18400 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 19:04:45.568961 18414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:45.570297 18400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:04:45.570470 18400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "d6e3444ae531472881c8c07e637e61ac"
format_stamp: "Formatted at 2026-09-24 19:04:45 on dist-test-slave-9fn1"
server_key: "5bdbf5160634a9180529684aa4a85902"
server_key_iv: "051ef0bbad1ae1b65ce293a29f8213cf"
server_key_version: "encryptionkey@0"
I20260924 19:04:45.570703 18400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:45.583582 18400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:45.584367 18400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:45.584643 18400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:45.585717 18400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:04:45.587210 18400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:04:45.587340 18400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:45.587476 18400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:04:45.587563 18400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:45.620076 18400 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43229
I20260924 19:04:45.620160 18526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43229 every 8 connection(s)
I20260924 19:04:45.621413 18400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:04:45.630395 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 18400
I20260924 19:04:45.630584 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-2/wal/instance
I20260924 19:04:45.631119 388 external_mini_cluster.cc:1428] Setting key 71f1df3c2c1e83322f0342608e827328
I20260924 19:04:45.634599 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:45.624231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40491 (local address 127.0.97.62:34955)
0924 19:04:45.624442 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:45.624448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:45.625896 (+ 1448us) server_negotiation.cc:407] Connection header received
0924 19:04:45.627575 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:45.627581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:45.627647 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:45.627733 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:45.628721 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.629440 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.631713 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.632269 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.632401 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:45.633808 (+ 1407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:45.633813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:45.633816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:04:45.633832 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:45.633861 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:04:45.633873 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:45.633969 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:45.634193 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 19:04:45.634284 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 19:04:45.635671 18529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:45.624810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34955 (local address 127.0.97.3:40491)
0924 19:04:45.625741 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:45.625781 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:45.627345 (+ 1564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:45.627929 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:45.627936 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:45.628018 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:04:45.628557 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:45.628572 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.629634 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.629639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:45.631564 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:45.631576 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.632498 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.632506 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:45.632731 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:45.633434 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:04:45.633437 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:04:45.633474 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:04:45.633621 (+ 147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:04:45.633968 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:04:45.633979 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:04:45.633990 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 19:04:45.634185 (+ 195us) client_negotiation.cc:240] Negotiation successful
0924 19:04:45.634559 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":282,"threads_started":1}
I20260924 19:04:45.636997 18527 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34955
I20260924 19:04:45.637284 18527 heartbeater.cc:464] Registering TS with master...
I20260924 19:04:45.637996 18527 heartbeater.cc:511] Master 127.0.97.62:34955 requested a full tablet report, sending...
I20260924 19:04:45.639263 18080 ts_manager.cc:194] Registered new tserver with Master: d6e3444ae531472881c8c07e637e61ac (127.0.97.3:43229)
I20260924 19:04:45.640053 18080 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:40491
I20260924 19:04:45.646071 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:04:45.655835 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:45.650147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51828 (local address 127.0.97.62:34955)
0924 19:04:45.650299 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:45.650303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:45.650807 (+ 504us) server_negotiation.cc:407] Connection header received
0924 19:04:45.650940 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:45.650944 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:45.651012 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:45.651093 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:45.651550 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.652232 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.653826 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.654200 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.654291 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:45.655229 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:45.655234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:45.655237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:04:45.655253 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:45.655283 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:04:45.655295 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:45.655362 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:45.655521 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 19:04:45.655596 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 19:04:45.659490 18080 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:04:45.666916 18080 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:04:45.697779 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:45.678984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52216 (local address 127.0.97.1:38141)
0924 19:04:45.690026 (+ 11042us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:45.690037 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:45.690083 (+ 46us) server_negotiation.cc:407] Connection header received
0924 19:04:45.690191 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:45.690202 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:45.690487 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:45.690717 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:45.691202 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.692673 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.695367 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.696496 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.696659 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:45.696900 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:45.697219 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 19:04:45.697376 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10847,"thread_start_us":11943,"threads_started":1}
I20260924 19:04:45.697824 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:45.678907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38141 (local address 127.0.0.1:52216)
0924 19:04:45.688876 (+ 9969us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:45.688896 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:45.689002 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:45.690731 (+ 1729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:45.690736 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:45.690767 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:45.691016 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:45.691024 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.692805 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.692813 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:45.695146 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:45.695160 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.696701 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.696708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:45.696969 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:45.697020 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 19:04:45.697173 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 19:04:45.697364 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9843,"mutex_wait_us":3,"thread_start_us":8985,"threads_started":1}
I20260924 19:04:45.701267 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:45.678811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54190 (local address 127.0.97.2:35925)
0924 19:04:45.691514 (+ 12703us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:45.691526 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:45.691556 (+ 30us) server_negotiation.cc:407] Connection header received
0924 19:04:45.691659 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:45.691669 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:45.691918 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:45.692129 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:45.693163 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.694766 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.697880 (+ 3114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.699653 (+ 1773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.699809 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:45.700153 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:45.700751 (+ 598us) server_negotiation.cc:299] Negotiation successful
0924 19:04:45.700907 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12527,"thread_start_us":12423,"threads_started":1}
I20260924 19:04:45.701344 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:45.678532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35925 (local address 127.0.0.1:54190)
0924 19:04:45.688324 (+ 9792us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:45.688372 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:45.688553 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:45.692433 (+ 3880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:45.692440 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:45.692474 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:45.692951 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:45.692971 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.694923 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.694930 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:45.697590 (+ 2660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:45.697626 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.700287 (+ 2661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.700296 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:45.700538 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:45.700579 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 19:04:45.700683 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 19:04:45.700853 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9534,"mutex_wait_us":70,"thread_start_us":10214,"threads_started":1}
I20260924 19:04:45.702091 18201 tablet_service.cc:1467] Processing CreateTablet for tablet 8e1e1e50fc884465a9c31049fe756d0c (DEFAULT_TABLE table=test-table [id=58d54ecde2584809b27b75f23ae3a99f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:04:45.703342 18201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e1e1e50fc884465a9c31049fe756d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:04:45.705506 18332 tablet_service.cc:1467] Processing CreateTablet for tablet 8e1e1e50fc884465a9c31049fe756d0c (DEFAULT_TABLE table=test-table [id=58d54ecde2584809b27b75f23ae3a99f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:04:45.707096 18332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e1e1e50fc884465a9c31049fe756d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:04:45.713387 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:45.681165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43229 (local address 127.0.0.1:44616)
0924 19:04:45.698030 (+ 16865us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:45.698047 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:45.698129 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:45.706641 (+ 8512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:45.706648 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:45.706673 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:45.706829 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:45.706838 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.708836 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.708842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:45.710860 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:45.710883 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.712619 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.712626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:45.712818 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:45.712854 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 19:04:45.712949 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 19:04:45.713089 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16772,"thread_start_us":25199,"threads_started":1}
I20260924 19:04:45.713572 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:45.682043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44616 (local address 127.0.97.3:43229)
0924 19:04:45.702236 (+ 20193us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:45.702247 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:45.702288 (+ 41us) server_negotiation.cc:407] Connection header received
0924 19:04:45.702402 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:45.705575 (+ 3173us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:45.705825 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:45.706012 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:45.707002 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.708681 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.711036 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.712425 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.712566 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:45.712830 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:45.713004 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 19:04:45.713185 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19989,"thread_start_us":19835,"threads_started":1}
I20260924 19:04:45.717976 18462 tablet_service.cc:1467] Processing CreateTablet for tablet 8e1e1e50fc884465a9c31049fe756d0c (DEFAULT_TABLE table=test-table [id=58d54ecde2584809b27b75f23ae3a99f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:04:45.718760 18545 tablet_bootstrap.cc:492] T 8e1e1e50fc884465a9c31049fe756d0c P 9aa402645fe24bd7813e0076a3aedb65: Bootstrap starting.
I20260924 19:04:45.719588 18462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e1e1e50fc884465a9c31049fe756d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:04:45.722232 18546 tablet_bootstrap.cc:492] T 8e1e1e50fc884465a9c31049fe756d0c P 0d99dd1c7b3e4e9489e6629b464b2c3f: Bootstrap starting.
I20260924 19:04:45.723163 18545 tablet_bootstrap.cc:654] T 8e1e1e50fc884465a9c31049fe756d0c P 9aa402645fe24bd7813e0076a3aedb65: Neither blocks nor log segments found. Creating new log.
I20260924 19:04:45.724164 18545 log.cc:824] T 8e1e1e50fc884465a9c31049fe756d0c P 9aa402645fe24bd7813e0076a3aedb65: Log is configured to *not* fsync() on all Append() calls
I20260924 19:04:45.725622 18546 tablet_bootstrap.cc:654] T 8e1e1e50fc884465a9c31049fe756d0c P 0d99dd1c7b3e4e9489e6629b464b2c3f: Neither blocks nor log segments found. Creating new log.
I20260924 19:04:45.726816 18546 log.cc:824] T 8e1e1e50fc884465a9c31049fe756d0c P 0d99dd1c7b3e4e9489e6629b464b2c3f: Log is configured to *not* fsync() on all Append() calls
I20260924 19:04:45.727624 18545 tablet_bootstrap.cc:492] T 8e1e1e50fc884465a9c31049fe756d0c P 9aa402645fe24bd7813e0076a3aedb65: No bootstrap required, opened a new log
I20260924 19:04:45.727916 18545 ts_tablet_manager.cc:1397] T 8e1e1e50fc884465a9c31049fe756d0c P 9aa402645fe24bd7813e0076a3aedb65: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260924 19:04:45.729825 18546 tablet_bootstrap.cc:492] T 8e1e1e50fc884465a9c31049fe756d0c P 0d99dd1c7b3e4e9489e6629b464b2c3f: No bootstrap required, opened a new log
I20260924 19:04:45.730077 18546 ts_tablet_manager.cc:1397] T 8e1e1e50fc884465a9c31049fe756d0c P 0d99dd1c7b3e4e9489e6629b464b2c3f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20260924 19:04:45.733901 18549 tablet_bootstrap.cc:492] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac: Bootstrap starting.
I20260924 19:04:45.736133 18545 raft_consensus.cc:348] T 8e1e1e50fc884465a9c31049fe756d0c P 9aa402645fe24bd7813e0076a3aedb65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d99dd1c7b3e4e9489e6629b464b2c3f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35925 } } peers { permanent_uuid: "9aa402645fe24bd7813e0076a3aedb65" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38141 } } peers { permanent_uuid: "d6e3444ae531472881c8c07e637e61ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43229 } }
I20260924 19:04:45.736586 18545 raft_consensus.cc:374] T 8e1e1e50fc884465a9c31049fe756d0c P 9aa402645fe24bd7813e0076a3aedb65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:04:45.736691 18545 raft_consensus.cc:729] T 8e1e1e50fc884465a9c31049fe756d0c P 9aa402645fe24bd7813e0076a3aedb65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aa402645fe24bd7813e0076a3aedb65, State: Initialized, Role: FOLLOWER
I20260924 19:04:45.737397 18545 consensus_queue.cc:260] T 8e1e1e50fc884465a9c31049fe756d0c P 9aa402645fe24bd7813e0076a3aedb65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d99dd1c7b3e4e9489e6629b464b2c3f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35925 } } peers { permanent_uuid: "9aa402645fe24bd7813e0076a3aedb65" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38141 } } peers { permanent_uuid: "d6e3444ae531472881c8c07e637e61ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43229 } }
I20260924 19:04:45.738485 18549 tablet_bootstrap.cc:654] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac: Neither blocks nor log segments found. Creating new log.
I20260924 19:04:45.738562 18266 heartbeater.cc:503] Master 127.0.97.62:34955 was elected leader, sending a full tablet report...
I20260924 19:04:45.739269 18545 ts_tablet_manager.cc:1428] T 8e1e1e50fc884465a9c31049fe756d0c P 9aa402645fe24bd7813e0076a3aedb65: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
I20260924 19:04:45.739807 18549 log.cc:824] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac: Log is configured to *not* fsync() on all Append() calls
I20260924 19:04:45.739645 18546 raft_consensus.cc:348] T 8e1e1e50fc884465a9c31049fe756d0c P 0d99dd1c7b3e4e9489e6629b464b2c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d99dd1c7b3e4e9489e6629b464b2c3f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35925 } } peers { permanent_uuid: "9aa402645fe24bd7813e0076a3aedb65" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38141 } } peers { permanent_uuid: "d6e3444ae531472881c8c07e637e61ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43229 } }
I20260924 19:04:45.740051 18546 raft_consensus.cc:374] T 8e1e1e50fc884465a9c31049fe756d0c P 0d99dd1c7b3e4e9489e6629b464b2c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:04:45.740181 18546 raft_consensus.cc:729] T 8e1e1e50fc884465a9c31049fe756d0c P 0d99dd1c7b3e4e9489e6629b464b2c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d99dd1c7b3e4e9489e6629b464b2c3f, State: Initialized, Role: FOLLOWER
I20260924 19:04:45.740751 18546 consensus_queue.cc:260] T 8e1e1e50fc884465a9c31049fe756d0c P 0d99dd1c7b3e4e9489e6629b464b2c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d99dd1c7b3e4e9489e6629b464b2c3f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35925 } } peers { permanent_uuid: "9aa402645fe24bd7813e0076a3aedb65" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38141 } } peers { permanent_uuid: "d6e3444ae531472881c8c07e637e61ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43229 } }
I20260924 19:04:45.741819 18397 heartbeater.cc:503] Master 127.0.97.62:34955 was elected leader, sending a full tablet report...
I20260924 19:04:45.742415 18546 ts_tablet_manager.cc:1428] T 8e1e1e50fc884465a9c31049fe756d0c P 0d99dd1c7b3e4e9489e6629b464b2c3f: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260924 19:04:45.745704 18549 tablet_bootstrap.cc:492] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac: No bootstrap required, opened a new log
I20260924 19:04:45.745970 18549 ts_tablet_manager.cc:1397] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260924 19:04:45.753404 18549 raft_consensus.cc:348] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d99dd1c7b3e4e9489e6629b464b2c3f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35925 } } peers { permanent_uuid: "9aa402645fe24bd7813e0076a3aedb65" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38141 } } peers { permanent_uuid: "d6e3444ae531472881c8c07e637e61ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43229 } }
I20260924 19:04:45.753859 18549 raft_consensus.cc:374] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:04:45.753969 18549 raft_consensus.cc:729] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e3444ae531472881c8c07e637e61ac, State: Initialized, Role: FOLLOWER
I20260924 19:04:45.754587 18549 consensus_queue.cc:260] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d99dd1c7b3e4e9489e6629b464b2c3f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35925 } } peers { permanent_uuid: "9aa402645fe24bd7813e0076a3aedb65" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38141 } } peers { permanent_uuid: "d6e3444ae531472881c8c07e637e61ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43229 } }
W20260924 19:04:45.755255 18267 tablet.cc:2369] T 8e1e1e50fc884465a9c31049fe756d0c P 9aa402645fe24bd7813e0076a3aedb65: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:04:45.755738 18527 heartbeater.cc:503] Master 127.0.97.62:34955 was elected leader, sending a full tablet report...
I20260924 19:04:45.756430 18549 ts_tablet_manager.cc:1428] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260924 19:04:45.842046 18553 raft_consensus.cc:482] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:04:45.842324 18553 raft_consensus.cc:504] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d99dd1c7b3e4e9489e6629b464b2c3f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35925 } } peers { permanent_uuid: "9aa402645fe24bd7813e0076a3aedb65" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38141 } } peers { permanent_uuid: "d6e3444ae531472881c8c07e637e61ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43229 } }
I20260924 19:04:45.843947 18553 leader_election.cc:288] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d99dd1c7b3e4e9489e6629b464b2c3f (127.0.97.2:35925), 9aa402645fe24bd7813e0076a3aedb65 (127.0.97.1:38141)
I20260924 19:04:45.850313 18529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:45.844286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38141 (local address 127.0.97.3:41961)
0924 19:04:45.844560 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:45.844583 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:45.844734 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:45.845182 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:45.845187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:45.845248 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:45.845435 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:45.845445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.846647 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.846652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:45.848423 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:45.848438 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.849771 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.849775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:45.849893 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:45.849917 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 19:04:45.850014 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 19:04:45.850107 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"spinlock_wait_cycles":3840}
I20260924 19:04:45.850467 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:45.844555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41961 (local address 127.0.97.1:38141)
0924 19:04:45.844807 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:45.844814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:45.844837 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:04:45.844914 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:45.844919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:45.845006 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:45.845115 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:45.845608 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.846511 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.848594 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.849638 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.849780 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:45.849873 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:45.850063 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 19:04:45.850190 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 19:04:45.851382 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:45.844283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39733 (local address 127.0.97.2:35925)
0924 19:04:45.844485 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:45.844491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:45.845186 (+ 695us) server_negotiation.cc:407] Connection header received
0924 19:04:45.845411 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:45.845416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:45.845494 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:45.845614 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:45.846130 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.846985 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.849045 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.850255 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.850387 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:45.850477 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:45.851078 (+ 601us) server_negotiation.cc:299] Negotiation successful
0924 19:04:45.851192 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 19:04:45.851533 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:45.844355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35925 (local address 127.0.97.3:39733)
0924 19:04:45.845069 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:45.845095 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:45.845268 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:45.845623 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:45.845628 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:45.845652 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:45.845973 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:45.845984 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.847234 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.847238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:45.848870 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:45.848882 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.850759 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.850766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:45.850892 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:45.850913 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 19:04:45.851038 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 19:04:45.851202 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"spinlock_wait_cycles":67200,"thread_start_us":383,"threads_started":1}
I20260924 19:04:45.851496 18221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e1e1e50fc884465a9c31049fe756d0c" candidate_uuid: "d6e3444ae531472881c8c07e637e61ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aa402645fe24bd7813e0076a3aedb65" is_pre_election: true
I20260924 19:04:45.852092 18221 raft_consensus.cc:2454] T 8e1e1e50fc884465a9c31049fe756d0c P 9aa402645fe24bd7813e0076a3aedb65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e3444ae531472881c8c07e637e61ac in term 0.
I20260924 19:04:45.852427 18352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e1e1e50fc884465a9c31049fe756d0c" candidate_uuid: "d6e3444ae531472881c8c07e637e61ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d99dd1c7b3e4e9489e6629b464b2c3f" is_pre_election: true
I20260924 19:04:45.852782 18418 leader_election.cc:302] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9aa402645fe24bd7813e0076a3aedb65, d6e3444ae531472881c8c07e637e61ac; no voters:
I20260924 19:04:45.853037 18352 raft_consensus.cc:2454] T 8e1e1e50fc884465a9c31049fe756d0c P 0d99dd1c7b3e4e9489e6629b464b2c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6e3444ae531472881c8c07e637e61ac in term 0.
I20260924 19:04:45.853220 18553 raft_consensus.cc:2781] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:04:45.853384 18553 raft_consensus.cc:482] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:04:45.853529 18553 raft_consensus.cc:3037] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:04:45.855721 18553 raft_consensus.cc:504] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d99dd1c7b3e4e9489e6629b464b2c3f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35925 } } peers { permanent_uuid: "9aa402645fe24bd7813e0076a3aedb65" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38141 } } peers { permanent_uuid: "d6e3444ae531472881c8c07e637e61ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43229 } }
I20260924 19:04:45.856282 18553 leader_election.cc:288] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac [CANDIDATE]: Term 1 election: Requested vote from peers 0d99dd1c7b3e4e9489e6629b464b2c3f (127.0.97.2:35925), 9aa402645fe24bd7813e0076a3aedb65 (127.0.97.1:38141)
I20260924 19:04:45.856726 18352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e1e1e50fc884465a9c31049fe756d0c" candidate_uuid: "d6e3444ae531472881c8c07e637e61ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d99dd1c7b3e4e9489e6629b464b2c3f"
I20260924 19:04:45.856740 18221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e1e1e50fc884465a9c31049fe756d0c" candidate_uuid: "d6e3444ae531472881c8c07e637e61ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9aa402645fe24bd7813e0076a3aedb65"
I20260924 19:04:45.856927 18352 raft_consensus.cc:3037] T 8e1e1e50fc884465a9c31049fe756d0c P 0d99dd1c7b3e4e9489e6629b464b2c3f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:04:45.856935 18221 raft_consensus.cc:3037] T 8e1e1e50fc884465a9c31049fe756d0c P 9aa402645fe24bd7813e0076a3aedb65 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:04:45.859110 18221 raft_consensus.cc:2454] T 8e1e1e50fc884465a9c31049fe756d0c P 9aa402645fe24bd7813e0076a3aedb65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e3444ae531472881c8c07e637e61ac in term 1.
I20260924 19:04:45.859169 18352 raft_consensus.cc:2454] T 8e1e1e50fc884465a9c31049fe756d0c P 0d99dd1c7b3e4e9489e6629b464b2c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6e3444ae531472881c8c07e637e61ac in term 1.
I20260924 19:04:45.859498 18418 leader_election.cc:302] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9aa402645fe24bd7813e0076a3aedb65, d6e3444ae531472881c8c07e637e61ac; no voters:
I20260924 19:04:45.859760 18553 raft_consensus.cc:2781] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:04:45.860244 18553 raft_consensus.cc:686] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac [term 1 LEADER]: Becoming Leader. State: Replica: d6e3444ae531472881c8c07e637e61ac, State: Running, Role: LEADER
I20260924 19:04:45.860740 18553 consensus_queue.cc:237] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d99dd1c7b3e4e9489e6629b464b2c3f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35925 } } peers { permanent_uuid: "9aa402645fe24bd7813e0076a3aedb65" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38141 } } peers { permanent_uuid: "d6e3444ae531472881c8c07e637e61ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43229 } }
I20260924 19:04:45.865576 18080 catalog_manager.cc:5495] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac reported cstate change: term changed from 0 to 1, leader changed from <none> to d6e3444ae531472881c8c07e637e61ac (127.0.97.3). New cstate: current_term: 1 leader_uuid: "d6e3444ae531472881c8c07e637e61ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d99dd1c7b3e4e9489e6629b464b2c3f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9aa402645fe24bd7813e0076a3aedb65" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6e3444ae531472881c8c07e637e61ac" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43229 } health_report { overall_health: HEALTHY } } }
W20260924 19:04:45.873016 18528 tablet.cc:2369] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:04:45.887298 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:45.882482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44618 (local address 127.0.97.3:43229)
0924 19:04:45.882660 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:45.882664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:45.882682 (+ 18us) server_negotiation.cc:407] Connection header received
0924 19:04:45.882846 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:45.882849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:45.882930 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:45.883045 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:04:45.883340 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.884047 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.885437 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.885870 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.885962 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:45.886362 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:04:45.886859 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:04:45.886963 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:45.887059 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 19:04:45.887117 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:04:45.894860 18352 raft_consensus.cc:1261] T 8e1e1e50fc884465a9c31049fe756d0c P 0d99dd1c7b3e4e9489e6629b464b2c3f [term 1 FOLLOWER]: Refusing update from remote peer d6e3444ae531472881c8c07e637e61ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:04:45.894896 18221 raft_consensus.cc:1261] T 8e1e1e50fc884465a9c31049fe756d0c P 9aa402645fe24bd7813e0076a3aedb65 [term 1 FOLLOWER]: Refusing update from remote peer d6e3444ae531472881c8c07e637e61ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:04:45.895800 18555 consensus_queue.cc:1035] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d99dd1c7b3e4e9489e6629b464b2c3f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:04:45.896132 18553 consensus_queue.cc:1035] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "9aa402645fe24bd7813e0076a3aedb65" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:04:45.908509 18558 mvcc.cc:205] Tried to move back new op lower bound from 7332973305415393280 to 7332973305289240576. Current Snapshot: MvccSnapshot[applied={T|T < 7332973305415393280}]
I20260924 19:04:45.920569 18079 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:04:45.920866 18079 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:04:45.924582 18079 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a465d0020f94af5964a0849126a0286: Sending DeleteTablet for 3 replicas of tablet 8e1e1e50fc884465a9c31049fe756d0c
I20260924 19:04:45.925871 18332 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e1e1e50fc884465a9c31049fe756d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:04:45 UTC) from {username='slave'} at 127.0.0.1:54190
I20260924 19:04:45.925894 18201 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e1e1e50fc884465a9c31049fe756d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:04:45 UTC) from {username='slave'} at 127.0.0.1:52216
I20260924 19:04:45.925953 18462 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e1e1e50fc884465a9c31049fe756d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:04:45 UTC) from {username='slave'} at 127.0.0.1:44616
I20260924 19:04:45.928038 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 18139
I20260924 19:04:45.935182 18569 tablet_replica.cc:330] T 8e1e1e50fc884465a9c31049fe756d0c P 9aa402645fe24bd7813e0076a3aedb65: stopping tablet replica
I20260924 19:04:45.937263 18569 raft_consensus.cc:2229] T 8e1e1e50fc884465a9c31049fe756d0c P 9aa402645fe24bd7813e0076a3aedb65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:04:45.937848 18569 raft_consensus.cc:2258] T 8e1e1e50fc884465a9c31049fe756d0c P 9aa402645fe24bd7813e0076a3aedb65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:04:45.944464 18569 ts_tablet_manager.cc:1905] T 8e1e1e50fc884465a9c31049fe756d0c P 9aa402645fe24bd7813e0076a3aedb65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:04:45.947575 18570 tablet_replica.cc:330] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac: stopping tablet replica
I20260924 19:04:45.948117 18570 raft_consensus.cc:2229] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:04:45.948956 18570 raft_consensus.cc:2258] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:04:45.951511 18569 ts_tablet_manager.cc:1918] T 8e1e1e50fc884465a9c31049fe756d0c P 9aa402645fe24bd7813e0076a3aedb65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:04:45.951689 18569 log.cc:1196] T 8e1e1e50fc884465a9c31049fe756d0c P 9aa402645fe24bd7813e0076a3aedb65: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-0/wal/wals/8e1e1e50fc884465a9c31049fe756d0c
I20260924 19:04:45.951967 18568 tablet_replica.cc:330] T 8e1e1e50fc884465a9c31049fe756d0c P 0d99dd1c7b3e4e9489e6629b464b2c3f: stopping tablet replica
I20260924 19:04:45.952236 18569 ts_tablet_manager.cc:1939] T 8e1e1e50fc884465a9c31049fe756d0c P 9aa402645fe24bd7813e0076a3aedb65: Deleting consensus metadata
I20260924 19:04:45.952440 18568 raft_consensus.cc:2229] T 8e1e1e50fc884465a9c31049fe756d0c P 0d99dd1c7b3e4e9489e6629b464b2c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:04:45.952883 18568 raft_consensus.cc:2258] T 8e1e1e50fc884465a9c31049fe756d0c P 0d99dd1c7b3e4e9489e6629b464b2c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:04:45.953251 18570 ts_tablet_manager.cc:1905] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:04:45.954233 18067 catalog_manager.cc:4841] TS 9aa402645fe24bd7813e0076a3aedb65 (127.0.97.1:38141): tablet 8e1e1e50fc884465a9c31049fe756d0c (table test-table [id=58d54ecde2584809b27b75f23ae3a99f]) successfully deleted
I20260924 19:04:45.957742 18542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:45.928816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52232 (local address 127.0.97.1:38141)
0924 19:04:45.929008 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:45.929017 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:45.935652 (+ 6635us) server_negotiation.cc:407] Connection header received
0924 19:04:45.936009 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:45.936015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:45.936133 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:45.936211 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:45.938279 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.939410 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.953634 (+ 14224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:45.954368 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:45.954502 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:45.955800 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:45.955810 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:45.955828 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:04:45.956567 (+ 739us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:45.956622 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:04:45.956646 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:45.956758 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:45.957265 (+ 507us) server_negotiation.cc:299] Negotiation successful
0924 19:04:45.957378 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 19:04:45.958352 18568 ts_tablet_manager.cc:1905] T 8e1e1e50fc884465a9c31049fe756d0c P 0d99dd1c7b3e4e9489e6629b464b2c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:04:45.958427 18261 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:04:45.961205 18570 ts_tablet_manager.cc:1918] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:04:45.961470 18570 log.cc:1196] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-2/wal/wals/8e1e1e50fc884465a9c31049fe756d0c
I20260924 19:04:45.962242 18570 ts_tablet_manager.cc:1939] T 8e1e1e50fc884465a9c31049fe756d0c P d6e3444ae531472881c8c07e637e61ac: Deleting consensus metadata
I20260924 19:04:45.964718 18065 catalog_manager.cc:4841] TS d6e3444ae531472881c8c07e637e61ac (127.0.97.3:43229): tablet 8e1e1e50fc884465a9c31049fe756d0c (table test-table [id=58d54ecde2584809b27b75f23ae3a99f]) successfully deleted
I20260924 19:04:45.965159 18568 ts_tablet_manager.cc:1918] T 8e1e1e50fc884465a9c31049fe756d0c P 0d99dd1c7b3e4e9489e6629b464b2c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:04:45.965314 18568 log.cc:1196] T 8e1e1e50fc884465a9c31049fe756d0c P 0d99dd1c7b3e4e9489e6629b464b2c3f: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276386554690-388-0/minicluster-data/ts-1/wal/wals/8e1e1e50fc884465a9c31049fe756d0c
I20260924 19:04:45.965850 18568 ts_tablet_manager.cc:1939] T 8e1e1e50fc884465a9c31049fe756d0c P 0d99dd1c7b3e4e9489e6629b464b2c3f: Deleting consensus metadata
I20260924 19:04:45.967941 18064 catalog_manager.cc:4841] TS 0d99dd1c7b3e4e9489e6629b464b2c3f (127.0.97.2:35925): tablet 8e1e1e50fc884465a9c31049fe756d0c (table test-table [id=58d54ecde2584809b27b75f23ae3a99f]) successfully deleted
I20260924 19:04:46.054795 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 18139
I20260924 19:04:46.073663 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 18270
I20260924 19:04:46.081022 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:46.074749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54192 (local address 127.0.97.2:35925)
0924 19:04:46.074950 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:46.074978 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:46.074997 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:04:46.075059 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:46.075064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:46.075154 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:46.075286 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:46.075673 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:46.076390 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:46.077887 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:46.078286 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:46.078469 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:46.079677 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:46.079686 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:46.079700 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:04:46.080296 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:46.080343 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:04:46.080361 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:46.080486 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:46.080590 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 19:04:46.080694 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 19:04:46.081843 18392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:04:46.170388 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 18270
I20260924 19:04:46.187851 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 18400
I20260924 19:04:46.195824 18544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:46.188656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44620 (local address 127.0.97.3:43229)
0924 19:04:46.188851 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:46.188858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:46.188881 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:04:46.189050 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:46.189055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:46.189170 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:46.189298 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:46.189754 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:46.190992 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:46.192218 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:46.192902 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:46.193056 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:46.194126 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:46.194136 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:46.194153 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:04:46.195003 (+ 850us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:46.195084 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:04:46.195111 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:46.195235 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:46.195318 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 19:04:46.195424 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 19:04:46.196630 18522 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:04:46.316895 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 18400
I20260924 19:04:46.336388 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 18049
I20260924 19:04:46.337423 18110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:04:46.443571 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 18049
2026-09-24T19:04:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2720 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:04:46 dist-test-slave-9fn1 krb5kdc[18578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:04:46 dist-test-slave-9fn1 krb5kdc[18578](info): set up 2 sockets
Sep 24 19:04:46 dist-test-slave-9fn1 krb5kdc[18578](info): commencing operation
krb5kdc: starting...
W20260924 19:04:47.553608 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:04:47 dist-test-slave-9fn1 krb5kdc[18578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:04:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:04:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 19:04:47.794634 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33541
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:04:48.120728 18591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:48.121025 18591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:48.121177 18591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:48.128345 18591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:04:48.128441 18591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:48.128523 18591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:48.128621 18591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:04:48.128679 18591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:04:48.138510 18591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:48.139734 18591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:48.141021 18591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:48.151978 18596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:48.152041 18599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:48.152063 18597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:48.153158 18591 server_base.cc:1034] running on GCE node
I20260924 19:04:48.153945 18591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:48.155339 18591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:48.156562 18591 hybrid_clock.cc:648] HybridClock initialized: now 1790276688156512 us; error 57 us; skew 500 ppm
I20260924 19:04:48.156889 18591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:04:48 dist-test-slave-9fn1 krb5kdc[18578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:04:48.160085 18591 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:04:48.161489 18591 webserver.cc:460] Webserver started at http://127.0.97.62:42313/ using document root <none> and password file <none>
I20260924 19:04:48.162235 18591 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:48.162387 18591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:48.162694 18591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:04:48.165735 18591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "43540fde65f640db982921ebf41941e4"
format_stamp: "Formatted at 2026-09-24 19:04:48 on dist-test-slave-9fn1"
server_key: "a33a267977c5016d8ee25d0a9b02b844"
server_key_iv: "8d1c9b831ab68d6e38c926ccb6a6150a"
server_key_version: "encryptionkey@0"
I20260924 19:04:48.166483 18591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "43540fde65f640db982921ebf41941e4"
format_stamp: "Formatted at 2026-09-24 19:04:48 on dist-test-slave-9fn1"
server_key: "a33a267977c5016d8ee25d0a9b02b844"
server_key_iv: "8d1c9b831ab68d6e38c926ccb6a6150a"
server_key_version: "encryptionkey@0"
I20260924 19:04:48.171638 18591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260924 19:04:48.175158 18606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:48.176507 18591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:04:48.176700 18591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "43540fde65f640db982921ebf41941e4"
format_stamp: "Formatted at 2026-09-24 19:04:48 on dist-test-slave-9fn1"
server_key: "a33a267977c5016d8ee25d0a9b02b844"
server_key_iv: "8d1c9b831ab68d6e38c926ccb6a6150a"
server_key_version: "encryptionkey@0"
I20260924 19:04:48.176911 18591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:48.200951 18591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:48.205456 18591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:48.205768 18591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:48.224035 18591 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33541
I20260924 19:04:48.224080 18657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33541 every 8 connection(s)
I20260924 19:04:48.225414 18591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:04:48.226502 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 18591
I20260924 19:04:48.226708 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:04:48.227401 388 external_mini_cluster.cc:1428] Setting key 89100c535def2b47a4c87720b128926e
I20260924 19:04:48.230124 18658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 19:04:48 dist-test-slave-9fn1 krb5kdc[18578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:04:48.243062 18658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43540fde65f640db982921ebf41941e4: Bootstrap starting.
I20260924 19:04:48.246476 18658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43540fde65f640db982921ebf41941e4: Neither blocks nor log segments found. Creating new log.
I20260924 19:04:48.247449 18658 log.cc:824] T 00000000000000000000000000000000 P 43540fde65f640db982921ebf41941e4: Log is configured to *not* fsync() on all Append() calls
I20260924 19:04:48.250592 18658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43540fde65f640db982921ebf41941e4: No bootstrap required, opened a new log
I20260924 19:04:48.250623 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:48.229299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42098 (local address 127.0.97.62:33541)
0924 19:04:48.230104 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:48.230118 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:48.230158 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:04:48.231999 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:48.232018 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:48.232365 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:48.232694 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:48.234475 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:48.235823 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:48.237684 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:48.238387 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:48.238528 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:48.243235 (+ 4707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:48.243245 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:48.243263 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:48.243303 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:48.246429 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:48.247132 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:48.247138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:48.247143 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:48.247312 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:48.247719 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:48.247723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:48.247725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:48.248234 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:48.248569 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:48.248989 (+ 420us) server_negotiation.cc:299] Negotiation successful
0924 19:04:48.249369 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":389,"threads_started":1}
I20260924 19:04:48.258663 18658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43540fde65f640db982921ebf41941e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43540fde65f640db982921ebf41941e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33541 } }
I20260924 19:04:48.259037 18658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43540fde65f640db982921ebf41941e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:04:48.259145 18658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43540fde65f640db982921ebf41941e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43540fde65f640db982921ebf41941e4, State: Initialized, Role: FOLLOWER
I20260924 19:04:48.259748 18658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43540fde65f640db982921ebf41941e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43540fde65f640db982921ebf41941e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33541 } }
I20260924 19:04:48.260006 18658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43540fde65f640db982921ebf41941e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:04:48.260175 18658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43540fde65f640db982921ebf41941e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:04:48.260366 18658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43540fde65f640db982921ebf41941e4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:04:48.262516 18658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43540fde65f640db982921ebf41941e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43540fde65f640db982921ebf41941e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33541 } }
I20260924 19:04:48.262995 18658 leader_election.cc:302] T 00000000000000000000000000000000 P 43540fde65f640db982921ebf41941e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43540fde65f640db982921ebf41941e4; no voters:
I20260924 19:04:48.263551 18658 leader_election.cc:288] T 00000000000000000000000000000000 P 43540fde65f640db982921ebf41941e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:04:48.263679 18663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43540fde65f640db982921ebf41941e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:04:48.264357 18663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43540fde65f640db982921ebf41941e4 [term 1 LEADER]: Becoming Leader. State: Replica: 43540fde65f640db982921ebf41941e4, State: Running, Role: LEADER
I20260924 19:04:48.264845 18663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43540fde65f640db982921ebf41941e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43540fde65f640db982921ebf41941e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33541 } }
I20260924 19:04:48.265607 18658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43540fde65f640db982921ebf41941e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:04:48.267511 18665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43540fde65f640db982921ebf41941e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43540fde65f640db982921ebf41941e4. Latest consensus state: current_term: 1 leader_uuid: "43540fde65f640db982921ebf41941e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43540fde65f640db982921ebf41941e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33541 } } }
I20260924 19:04:48.267935 18665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43540fde65f640db982921ebf41941e4 [sys.catalog]: This master's current role is: LEADER
I20260924 19:04:48.267895 18664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43540fde65f640db982921ebf41941e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43540fde65f640db982921ebf41941e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43540fde65f640db982921ebf41941e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33541 } } }
I20260924 19:04:48.268224 18664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43540fde65f640db982921ebf41941e4 [sys.catalog]: This master's current role is: LEADER
I20260924 19:04:48.270422 18670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:04:48.274936 18670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:04:48.282989 18670 catalog_manager.cc:1347] Generated new cluster ID: 7ad9d3bce3984e658eec4e2a6667224e
I20260924 19:04:48.283103 18670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:04:48.314508 18670 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:04:48.316100 18670 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:04:48.326388 18670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43540fde65f640db982921ebf41941e4: Generated new TSK 0
I20260924 19:04:48.327270 18670 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 19:04:48.381317 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33541
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:04:48.709347 18684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:48.709679 18684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:48.709873 18684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:48.717485 18684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:48.717633 18684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:48.717810 18684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:04:48.727608 18684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:48.728950 18684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:48.730325 18684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:48.743898 18692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:48.743960 18690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:48.743777 18689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:48.744747 18684 server_base.cc:1034] running on GCE node
I20260924 19:04:48.745368 18684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:48.746327 18684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:48.747586 18684 hybrid_clock.cc:648] HybridClock initialized: now 1790276688747543 us; error 55 us; skew 500 ppm
I20260924 19:04:48.747975 18684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:04:48 dist-test-slave-9fn1 krb5kdc[18578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:04:48.751313 18684 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 19:04:48.752790 18684 webserver.cc:460] Webserver started at http://127.0.97.1:46865/ using document root <none> and password file <none>
I20260924 19:04:48.753448 18684 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:48.753620 18684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:48.753891 18684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:04:48.756745 18684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-0/data/instance:
uuid: "f1cba60563724e79b1bd286b94752786"
format_stamp: "Formatted at 2026-09-24 19:04:48 on dist-test-slave-9fn1"
server_key: "21f47705df0847576bd892a3fa69a3b9"
server_key_iv: "046fb30b9a7ac55c444729f7afcb260a"
server_key_version: "encryptionkey@0"
I20260924 19:04:48.757629 18684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-0/wal/instance:
uuid: "f1cba60563724e79b1bd286b94752786"
format_stamp: "Formatted at 2026-09-24 19:04:48 on dist-test-slave-9fn1"
server_key: "21f47705df0847576bd892a3fa69a3b9"
server_key_iv: "046fb30b9a7ac55c444729f7afcb260a"
server_key_version: "encryptionkey@0"
I20260924 19:04:48.762748 18684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 19:04:48.766152 18699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:48.767640 18684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:04:48.767800 18684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "f1cba60563724e79b1bd286b94752786"
format_stamp: "Formatted at 2026-09-24 19:04:48 on dist-test-slave-9fn1"
server_key: "21f47705df0847576bd892a3fa69a3b9"
server_key_iv: "046fb30b9a7ac55c444729f7afcb260a"
server_key_version: "encryptionkey@0"
I20260924 19:04:48.768009 18684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:48.790086 18684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:48.794263 18684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:48.794543 18684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:48.795495 18684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:04:48.796804 18684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:04:48.796880 18684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:48.797043 18684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:04:48.797102 18684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:48.830760 18684 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37933
I20260924 19:04:48.830654 18811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37933 every 8 connection(s)
I20260924 19:04:48.832818 18684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:04:48.838482 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 18684
I20260924 19:04:48.838673 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-0/wal/instance
I20260924 19:04:48.839241 388 external_mini_cluster.cc:1428] Setting key 0bde5d2ff5226d7d41f2b889d0438993
Sep 24 19:04:48 dist-test-slave-9fn1 krb5kdc[18578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:04:48.862782 18815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:48.837018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33541 (local address 127.0.97.1:33215)
0924 19:04:48.837831 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:48.837893 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:48.840373 (+ 2480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:48.841130 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:48.841141 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:48.841825 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:04:48.842641 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:48.842668 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:48.843750 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:48.843756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:48.845844 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:48.845863 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:48.846956 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:48.846965 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:48.847282 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:48.847996 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:04:48.848029 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:04:48.853240 (+ 5211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:04:48.857642 (+ 4402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:04:48.857653 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:04:48.857675 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:04:48.858360 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:04:48.858812 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:04:48.858819 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:04:48.858826 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:04:48.859077 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:04:48.859832 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:04:48.859843 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:04:48.860310 (+ 467us) client_negotiation.cc:769] Sending connection context
0924 19:04:48.860613 (+ 303us) client_negotiation.cc:240] Negotiation successful
0924 19:04:48.861221 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":958,"threads_started":1}
I20260924 19:04:48.864104 18814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:48.836478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33215 (local address 127.0.97.62:33541)
0924 19:04:48.836898 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:48.836905 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:48.838040 (+ 1135us) server_negotiation.cc:407] Connection header received
0924 19:04:48.840648 (+ 2608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:48.840656 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:48.840743 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:48.840875 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:48.842833 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:48.843600 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:48.846047 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:48.846707 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:48.846854 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:48.853568 (+ 6714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:48.853577 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:48.853583 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:48.853632 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:48.857422 (+ 3790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:48.858510 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:48.858518 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:48.858524 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:48.858672 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:48.859227 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:48.859234 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:48.859240 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:48.859614 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:48.859761 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:48.863373 (+ 3612us) server_negotiation.cc:299] Negotiation successful
0924 19:04:48.863636 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":192,"threads_started":1}
I20260924 19:04:48.865659 18812 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33541
I20260924 19:04:48.866045 18812 heartbeater.cc:464] Registering TS with master...
I20260924 19:04:48.866935 18812 heartbeater.cc:511] Master 127.0.97.62:33541 requested a full tablet report, sending...
I20260924 19:04:48.869434 18623 ts_manager.cc:194] Registered new tserver with Master: f1cba60563724e79b1bd286b94752786 (127.0.97.1:37933)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 19:04:48.871444 18623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33215
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 19:04:48.909009 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33541
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:04:49.231047 18818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:49.231284 18818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:49.231451 18818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:49.238453 18818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:49.238551 18818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:49.238731 18818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:04:49.248018 18818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:49.249217 18818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:49.250502 18818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:49.263441 18824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:49.263509 18826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:49.263486 18823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:49.264184 18818 server_base.cc:1034] running on GCE node
I20260924 19:04:49.264731 18818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:49.265894 18818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:49.267103 18818 hybrid_clock.cc:648] HybridClock initialized: now 1790276689267047 us; error 43 us; skew 500 ppm
I20260924 19:04:49.267374 18818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:04:49 dist-test-slave-9fn1 krb5kdc[18578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:04:49.275499 18818 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 19:04:49.276878 18818 webserver.cc:460] Webserver started at http://127.0.97.2:39887/ using document root <none> and password file <none>
I20260924 19:04:49.277585 18818 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:49.277693 18818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:49.277948 18818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:04:49.280817 18818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-1/data/instance:
uuid: "57036440df74406c9d3aa5fa0fa8b436"
format_stamp: "Formatted at 2026-09-24 19:04:49 on dist-test-slave-9fn1"
server_key: "67e4ab31b27e30a615e1fdc19fd2aa15"
server_key_iv: "5248c92a581edc3bba83118c952a40b0"
server_key_version: "encryptionkey@0"
I20260924 19:04:49.281631 18818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-1/wal/instance:
uuid: "57036440df74406c9d3aa5fa0fa8b436"
format_stamp: "Formatted at 2026-09-24 19:04:49 on dist-test-slave-9fn1"
server_key: "67e4ab31b27e30a615e1fdc19fd2aa15"
server_key_iv: "5248c92a581edc3bba83118c952a40b0"
server_key_version: "encryptionkey@0"
I20260924 19:04:49.286569 18818 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 19:04:49.289716 18833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:49.290959 18818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 19:04:49.291139 18818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "57036440df74406c9d3aa5fa0fa8b436"
format_stamp: "Formatted at 2026-09-24 19:04:49 on dist-test-slave-9fn1"
server_key: "67e4ab31b27e30a615e1fdc19fd2aa15"
server_key_iv: "5248c92a581edc3bba83118c952a40b0"
server_key_version: "encryptionkey@0"
I20260924 19:04:49.291328 18818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:49.343853 18818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:49.347856 18818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:49.348161 18818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:49.349154 18818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:04:49.350477 18818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:04:49.350581 18818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:49.350733 18818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:04:49.350818 18818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:49.383384 18818 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45525
I20260924 19:04:49.383468 18945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45525 every 8 connection(s)
I20260924 19:04:49.384655 18818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:04:49.392704 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 18818
I20260924 19:04:49.392920 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-1/wal/instance
I20260924 19:04:49.393374 388 external_mini_cluster.cc:1428] Setting key 4dce811b98541a8c3fcbd7ebb5f8803f
Sep 24 19:04:49 dist-test-slave-9fn1 krb5kdc[18578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:04:49.405143 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:49.387142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49541 (local address 127.0.97.62:33541)
0924 19:04:49.387563 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:49.387567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:49.388385 (+ 818us) server_negotiation.cc:407] Connection header received
0924 19:04:49.390233 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:49.390238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:49.390328 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:49.390442 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:49.391856 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:49.392754 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:49.394742 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:49.395397 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:49.395515 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:49.399518 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:49.399522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:49.399526 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:49.399556 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:49.402207 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:49.402937 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:49.402940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:49.402943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:49.403073 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:49.403555 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:49.403559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:49.403561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:49.403801 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:49.403906 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:49.404734 (+ 828us) server_negotiation.cc:299] Negotiation successful
0924 19:04:49.404888 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":232,"threads_started":1}
I20260924 19:04:49.406710 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:49.387600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33541 (local address 127.0.97.2:49541)
0924 19:04:49.388234 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:49.388269 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:49.389970 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:49.390673 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:49.390681 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:49.391160 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:04:49.391692 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:49.391709 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:49.392950 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:49.392979 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:49.394617 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:49.394629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:49.395631 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:49.395638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:49.395933 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:49.396633 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:04:49.396667 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:04:49.399296 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:04:49.402348 (+ 3052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:04:49.402357 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:04:49.402372 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:04:49.402799 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:04:49.403183 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:04:49.403191 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:04:49.403196 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:04:49.403442 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:04:49.403928 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:04:49.403936 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:04:49.404367 (+ 431us) client_negotiation.cc:769] Sending connection context
0924 19:04:49.404665 (+ 298us) client_negotiation.cc:240] Negotiation successful
0924 19:04:49.405234 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":242,"threads_started":1}
I20260924 19:04:49.408125 18946 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33541
I20260924 19:04:49.408430 18946 heartbeater.cc:464] Registering TS with master...
I20260924 19:04:49.409142 18946 heartbeater.cc:511] Master 127.0.97.62:33541 requested a full tablet report, sending...
I20260924 19:04:49.410604 18623 ts_manager.cc:194] Registered new tserver with Master: 57036440df74406c9d3aa5fa0fa8b436 (127.0.97.2:45525)
I20260924 19:04:49.411396 18623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49541
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 19:04:49.448287 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33541
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:04:49.767423 18952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:49.767750 18952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:49.767928 18952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:49.776328 18952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:49.776513 18952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:49.776728 18952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:04:49.787027 18952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:49.788328 18952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:49.789726 18952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:49.803509 18960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:49.803541 18958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:49.803469 18957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:49.804387 18952 server_base.cc:1034] running on GCE node
I20260924 19:04:49.805024 18952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:49.805992 18952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:49.807189 18952 hybrid_clock.cc:648] HybridClock initialized: now 1790276689807144 us; error 35 us; skew 500 ppm
I20260924 19:04:49.807504 18952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:04:49 dist-test-slave-9fn1 krb5kdc[18578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:04:49.810791 18952 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 19:04:49.812280 18952 webserver.cc:460] Webserver started at http://127.0.97.3:37359/ using document root <none> and password file <none>
I20260924 19:04:49.812971 18952 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:49.813117 18952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:49.813432 18952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:04:49.816354 18952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-2/data/instance:
uuid: "6f0e0296a4404693891b4746efd56c10"
format_stamp: "Formatted at 2026-09-24 19:04:49 on dist-test-slave-9fn1"
server_key: "5cd16ce9e267e271a9d8b7ae89a26096"
server_key_iv: "7abc62ccd68a518fbf7732344bb7f681"
server_key_version: "encryptionkey@0"
I20260924 19:04:49.817147 18952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6f0e0296a4404693891b4746efd56c10"
format_stamp: "Formatted at 2026-09-24 19:04:49 on dist-test-slave-9fn1"
server_key: "5cd16ce9e267e271a9d8b7ae89a26096"
server_key_iv: "7abc62ccd68a518fbf7732344bb7f681"
server_key_version: "encryptionkey@0"
I20260924 19:04:49.822261 18952 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 19:04:49.825445 18967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:49.826712 18952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 19:04:49.826896 18952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "6f0e0296a4404693891b4746efd56c10"
format_stamp: "Formatted at 2026-09-24 19:04:49 on dist-test-slave-9fn1"
server_key: "5cd16ce9e267e271a9d8b7ae89a26096"
server_key_iv: "7abc62ccd68a518fbf7732344bb7f681"
server_key_version: "encryptionkey@0"
I20260924 19:04:49.827097 18952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:49.839795 18952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:49.843976 18952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:49.844270 18952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:49.845156 18952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:04:49.846453 18952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:04:49.846551 18952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:49.846701 18952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:04:49.846772 18952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:49.875056 18812 heartbeater.cc:503] Master 127.0.97.62:33541 was elected leader, sending a full tablet report...
I20260924 19:04:49.881939 18952 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41013
I20260924 19:04:49.882020 19079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41013 every 8 connection(s)
I20260924 19:04:49.883210 18952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:04:49.890936 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 18952
I20260924 19:04:49.891139 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-2/wal/instance
I20260924 19:04:49.891569 388 external_mini_cluster.cc:1428] Setting key 76fb46c3c84dc85b83f29d84a3884abc
Sep 24 19:04:49 dist-test-slave-9fn1 krb5kdc[18578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:04:49.903081 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:49.885893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60097 (local address 127.0.97.62:33541)
0924 19:04:49.886071 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:49.886076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:49.887265 (+ 1189us) server_negotiation.cc:407] Connection header received
0924 19:04:49.889089 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:49.889095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:49.889199 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:49.889312 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:49.890934 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:49.891830 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:49.893937 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:49.894531 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:49.894630 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:49.897972 (+ 3342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:49.897976 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:49.897979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:49.898010 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:49.900426 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:49.901096 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:49.901099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:49.901102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:49.901210 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:49.901639 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:49.901642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:49.901645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:49.901927 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:49.902043 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:49.902692 (+ 649us) server_negotiation.cc:299] Negotiation successful
0924 19:04:49.902860 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 19:04:49.904026 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:49.886413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33541 (local address 127.0.97.3:60097)
0924 19:04:49.887096 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:49.887132 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:49.888827 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:49.889586 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:49.889594 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:49.890101 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:04:49.890745 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:49.890769 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:49.891995 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:49.892000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:49.893799 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:49.893816 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:49.894724 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:49.894730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:49.894919 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:49.895561 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:04:49.895582 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:04:49.897749 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:04:49.900544 (+ 2795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:04:49.900550 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:04:49.900562 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:04:49.900992 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:04:49.901297 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:04:49.901300 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:04:49.901303 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:04:49.901470 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:04:49.902035 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:04:49.902040 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:04:49.902347 (+ 307us) client_negotiation.cc:769] Sending connection context
0924 19:04:49.902564 (+ 217us) client_negotiation.cc:240] Negotiation successful
0924 19:04:49.902925 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":273,"threads_started":1}
I20260924 19:04:49.905567 19080 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33541
I20260924 19:04:49.905886 19080 heartbeater.cc:464] Registering TS with master...
I20260924 19:04:49.906528 19080 heartbeater.cc:511] Master 127.0.97.62:33541 requested a full tablet report, sending...
I20260924 19:04:49.907953 18623 ts_manager.cc:194] Registered new tserver with Master: 6f0e0296a4404693891b4746efd56c10 (127.0.97.3:41013)
I20260924 19:04:49.908639 18623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60097
I20260924 19:04:49.918753 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:04:49.935434 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:49.922804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42106 (local address 127.0.97.62:33541)
0924 19:04:49.922971 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:49.922976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:49.923457 (+ 481us) server_negotiation.cc:407] Connection header received
0924 19:04:49.923515 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:49.923518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:49.923580 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:49.923681 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:49.925015 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:49.925838 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:49.927176 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:49.927565 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:49.927655 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:49.930392 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:49.930399 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:49.930403 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:49.930449 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:49.933000 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:49.933794 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:49.933798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:49.933800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:49.933914 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:49.934261 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:49.934264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:49.934266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:49.934550 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:49.934747 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:49.935049 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 19:04:49.935177 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 19:04:49.939260 18623 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:04:49.946414 18623 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:04:49.969553 19092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:49.958839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41013 (local address 127.0.0.1:34892)
0924 19:04:49.960220 (+ 1381us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:49.960298 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:49.960515 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:49.961697 (+ 1182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:49.961708 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:49.961759 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:49.962314 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:49.962329 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:49.964862 (+ 2533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:49.964870 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:49.967142 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:49.967155 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:49.968430 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:49.968436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:49.968681 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:49.968741 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 19:04:49.968869 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 19:04:49.969057 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1066,"mutex_wait_us":45,"thread_start_us":905,"threads_started":1}
I20260924 19:04:49.970403 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:49.958912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34892 (local address 127.0.97.3:41013)
0924 19:04:49.959571 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:49.959583 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:49.960441 (+ 858us) server_negotiation.cc:407] Connection header received
0924 19:04:49.961059 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:49.961070 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:49.961380 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:49.962560 (+ 1180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:49.962927 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:49.964740 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:49.967303 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:49.968233 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:49.968376 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:49.968611 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:49.969871 (+ 1260us) server_negotiation.cc:299] Negotiation successful
0924 19:04:49.970052 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":304,"threads_started":1}
I20260924 19:04:49.972030 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:49.962156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45525 (local address 127.0.0.1:42388)
0924 19:04:49.962773 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:49.962799 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:49.963000 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:49.963811 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:49.963814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:49.963832 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:49.964077 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:49.964083 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:49.966471 (+ 2388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:49.966477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:49.968735 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:49.968748 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:49.971185 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:49.971199 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:49.971371 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:49.971425 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 19:04:49.971515 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 19:04:49.971654 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":69,"thread_start_us":384,"threads_started":1}
I20260924 19:04:49.972767 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:49.961821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42388 (local address 127.0.97.2:45525)
0924 19:04:49.962367 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:49.962374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:49.962900 (+ 526us) server_negotiation.cc:407] Connection header received
0924 19:04:49.963206 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:49.963216 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:49.963426 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:49.963617 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:49.964217 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:49.965947 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:49.968885 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:49.970985 (+ 2100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:49.971124 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:49.971317 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:49.972250 (+ 933us) server_negotiation.cc:299] Negotiation successful
0924 19:04:49.972419 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":449,"threads_started":1}
I20260924 19:04:49.974498 19015 tablet_service.cc:1467] Processing CreateTablet for tablet 67dedce8eb54427d892af94b5b3829b3 (DEFAULT_TABLE table=test-table [id=40a26d58527448b3bb2b106c5c176842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:04:49.976003 19015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67dedce8eb54427d892af94b5b3829b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:04:49.976125 18881 tablet_service.cc:1467] Processing CreateTablet for tablet 67dedce8eb54427d892af94b5b3829b3 (DEFAULT_TABLE table=test-table [id=40a26d58527448b3bb2b106c5c176842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:04:49.977385 18881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67dedce8eb54427d892af94b5b3829b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:04:49.978344 19094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:49.958988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37933 (local address 127.0.0.1:53234)
0924 19:04:49.961405 (+ 2417us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:49.961429 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:49.961553 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:49.965185 (+ 3632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:49.965189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:49.965212 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:49.965623 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:49.965635 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:49.967465 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:49.967472 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:49.969370 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:49.969387 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:49.977638 (+ 8251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:49.977643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:49.977762 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:49.977786 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 19:04:49.977893 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 19:04:49.978015 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2288,"thread_start_us":2165,"threads_started":1}
I20260924 19:04:49.978402 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:49.959287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53234 (local address 127.0.97.1:37933)
0924 19:04:49.964688 (+ 5401us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:49.964696 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:49.964722 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:04:49.964806 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:49.964813 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:49.964995 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:49.965174 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:49.965787 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:49.967329 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:49.969615 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:49.971276 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:49.971514 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:49.971799 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:49.977947 (+ 6148us) server_negotiation.cc:299] Negotiation successful
0924 19:04:49.978109 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5234,"thread_start_us":5085,"threads_started":1}
I20260924 19:04:49.981882 18747 tablet_service.cc:1467] Processing CreateTablet for tablet 67dedce8eb54427d892af94b5b3829b3 (DEFAULT_TABLE table=test-table [id=40a26d58527448b3bb2b106c5c176842]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:04:49.982971 18747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67dedce8eb54427d892af94b5b3829b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:04:49.990855 19098 tablet_bootstrap.cc:492] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10: Bootstrap starting.
I20260924 19:04:49.994830 19100 tablet_bootstrap.cc:492] T 67dedce8eb54427d892af94b5b3829b3 P f1cba60563724e79b1bd286b94752786: Bootstrap starting.
I20260924 19:04:49.995314 19098 tablet_bootstrap.cc:654] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10: Neither blocks nor log segments found. Creating new log.
I20260924 19:04:49.996129 19099 tablet_bootstrap.cc:492] T 67dedce8eb54427d892af94b5b3829b3 P 57036440df74406c9d3aa5fa0fa8b436: Bootstrap starting.
I20260924 19:04:49.996770 19098 log.cc:824] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10: Log is configured to *not* fsync() on all Append() calls
I20260924 19:04:49.999186 19100 tablet_bootstrap.cc:654] T 67dedce8eb54427d892af94b5b3829b3 P f1cba60563724e79b1bd286b94752786: Neither blocks nor log segments found. Creating new log.
I20260924 19:04:49.999366 19099 tablet_bootstrap.cc:654] T 67dedce8eb54427d892af94b5b3829b3 P 57036440df74406c9d3aa5fa0fa8b436: Neither blocks nor log segments found. Creating new log.
I20260924 19:04:50.000061 19098 tablet_bootstrap.cc:492] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10: No bootstrap required, opened a new log
I20260924 19:04:50.000239 19100 log.cc:824] T 67dedce8eb54427d892af94b5b3829b3 P f1cba60563724e79b1bd286b94752786: Log is configured to *not* fsync() on all Append() calls
I20260924 19:04:50.000340 19098 ts_tablet_manager.cc:1397] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260924 19:04:50.000553 19099 log.cc:824] T 67dedce8eb54427d892af94b5b3829b3 P 57036440df74406c9d3aa5fa0fa8b436: Log is configured to *not* fsync() on all Append() calls
I20260924 19:04:50.007966 19100 tablet_bootstrap.cc:492] T 67dedce8eb54427d892af94b5b3829b3 P f1cba60563724e79b1bd286b94752786: No bootstrap required, opened a new log
I20260924 19:04:50.008256 19100 ts_tablet_manager.cc:1397] T 67dedce8eb54427d892af94b5b3829b3 P f1cba60563724e79b1bd286b94752786: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20260924 19:04:50.008780 19099 tablet_bootstrap.cc:492] T 67dedce8eb54427d892af94b5b3829b3 P 57036440df74406c9d3aa5fa0fa8b436: No bootstrap required, opened a new log
I20260924 19:04:50.009045 19099 ts_tablet_manager.cc:1397] T 67dedce8eb54427d892af94b5b3829b3 P 57036440df74406c9d3aa5fa0fa8b436: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260924 19:04:50.012158 19098 raft_consensus.cc:348] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f0e0296a4404693891b4746efd56c10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41013 } } peers { permanent_uuid: "f1cba60563724e79b1bd286b94752786" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37933 } } peers { permanent_uuid: "57036440df74406c9d3aa5fa0fa8b436" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45525 } }
I20260924 19:04:50.012606 19098 raft_consensus.cc:374] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:04:50.012719 19098 raft_consensus.cc:729] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f0e0296a4404693891b4746efd56c10, State: Initialized, Role: FOLLOWER
I20260924 19:04:50.013587 19098 consensus_queue.cc:260] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f0e0296a4404693891b4746efd56c10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41013 } } peers { permanent_uuid: "f1cba60563724e79b1bd286b94752786" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37933 } } peers { permanent_uuid: "57036440df74406c9d3aa5fa0fa8b436" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45525 } }
I20260924 19:04:50.014693 19080 heartbeater.cc:503] Master 127.0.97.62:33541 was elected leader, sending a full tablet report...
I20260924 19:04:50.016636 19098 ts_tablet_manager.cc:1428] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260924 19:04:50.018299 19100 raft_consensus.cc:348] T 67dedce8eb54427d892af94b5b3829b3 P f1cba60563724e79b1bd286b94752786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f0e0296a4404693891b4746efd56c10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41013 } } peers { permanent_uuid: "f1cba60563724e79b1bd286b94752786" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37933 } } peers { permanent_uuid: "57036440df74406c9d3aa5fa0fa8b436" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45525 } }
I20260924 19:04:50.018733 19100 raft_consensus.cc:374] T 67dedce8eb54427d892af94b5b3829b3 P f1cba60563724e79b1bd286b94752786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:04:50.018550 19099 raft_consensus.cc:348] T 67dedce8eb54427d892af94b5b3829b3 P 57036440df74406c9d3aa5fa0fa8b436 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f0e0296a4404693891b4746efd56c10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41013 } } peers { permanent_uuid: "f1cba60563724e79b1bd286b94752786" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37933 } } peers { permanent_uuid: "57036440df74406c9d3aa5fa0fa8b436" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45525 } }
I20260924 19:04:50.018838 19100 raft_consensus.cc:729] T 67dedce8eb54427d892af94b5b3829b3 P f1cba60563724e79b1bd286b94752786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1cba60563724e79b1bd286b94752786, State: Initialized, Role: FOLLOWER
I20260924 19:04:50.018957 19099 raft_consensus.cc:374] T 67dedce8eb54427d892af94b5b3829b3 P 57036440df74406c9d3aa5fa0fa8b436 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:04:50.019081 19099 raft_consensus.cc:729] T 67dedce8eb54427d892af94b5b3829b3 P 57036440df74406c9d3aa5fa0fa8b436 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57036440df74406c9d3aa5fa0fa8b436, State: Initialized, Role: FOLLOWER
I20260924 19:04:50.019480 19100 consensus_queue.cc:260] T 67dedce8eb54427d892af94b5b3829b3 P f1cba60563724e79b1bd286b94752786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f0e0296a4404693891b4746efd56c10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41013 } } peers { permanent_uuid: "f1cba60563724e79b1bd286b94752786" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37933 } } peers { permanent_uuid: "57036440df74406c9d3aa5fa0fa8b436" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45525 } }
I20260924 19:04:50.019783 19099 consensus_queue.cc:260] T 67dedce8eb54427d892af94b5b3829b3 P 57036440df74406c9d3aa5fa0fa8b436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f0e0296a4404693891b4746efd56c10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41013 } } peers { permanent_uuid: "f1cba60563724e79b1bd286b94752786" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37933 } } peers { permanent_uuid: "57036440df74406c9d3aa5fa0fa8b436" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45525 } }
I20260924 19:04:50.021041 18946 heartbeater.cc:503] Master 127.0.97.62:33541 was elected leader, sending a full tablet report...
I20260924 19:04:50.021634 19099 ts_tablet_manager.cc:1428] T 67dedce8eb54427d892af94b5b3829b3 P 57036440df74406c9d3aa5fa0fa8b436: Time spent starting tablet: real 0.012s user 0.010s sys 0.004s
I20260924 19:04:50.022109 19100 ts_tablet_manager.cc:1428] T 67dedce8eb54427d892af94b5b3829b3 P f1cba60563724e79b1bd286b94752786: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260924 19:04:50.044854 19104 raft_consensus.cc:482] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:04:50.045176 19104 raft_consensus.cc:504] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f0e0296a4404693891b4746efd56c10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41013 } } peers { permanent_uuid: "f1cba60563724e79b1bd286b94752786" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37933 } } peers { permanent_uuid: "57036440df74406c9d3aa5fa0fa8b436" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45525 } }
I20260924 19:04:50.046877 19104 leader_election.cc:288] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1cba60563724e79b1bd286b94752786 (127.0.97.1:37933), 57036440df74406c9d3aa5fa0fa8b436 (127.0.97.2:45525)
I20260924 19:04:50.052377 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:50.047292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45505 (local address 127.0.97.1:37933)
0924 19:04:50.047488 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:50.047493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:50.047511 (+ 18us) server_negotiation.cc:407] Connection header received
0924 19:04:50.047699 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:50.047705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:50.047773 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:50.047851 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:50.048374 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:50.049049 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:50.050694 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:50.051489 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:50.051630 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:50.051728 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:50.052001 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 19:04:50.052127 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:04:50.052371 19082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:50.047188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37933 (local address 127.0.97.3:45505)
0924 19:04:50.047401 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:50.047422 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:50.047543 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:50.047903 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:50.047908 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:50.047944 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:50.048166 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:50.048180 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:50.049193 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:50.049200 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:50.050537 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:50.050549 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:50.051654 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:50.051660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:50.051841 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:50.051866 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 19:04:50.051966 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 19:04:50.052095 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 19:04:50.053366 19107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:50.047626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45525 (local address 127.0.97.3:48991)
0924 19:04:50.048337 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:50.048355 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:50.048472 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:50.048791 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:50.048795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:50.048819 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:50.049139 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:50.049148 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:50.050206 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:50.050211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:50.051549 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:50.051560 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:50.052735 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:50.052744 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:50.052918 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:50.052954 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 19:04:50.053054 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 19:04:50.053137 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":528,"threads_started":1}
I20260924 19:04:50.053476 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:50.048095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48991 (local address 127.0.97.2:45525)
0924 19:04:50.048324 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:50.048330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:50.048419 (+ 89us) server_negotiation.cc:407] Connection header received
0924 19:04:50.048615 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:50.048620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:50.048686 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:50.048777 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:50.049292 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:50.050060 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:50.051703 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:50.052479 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:50.052620 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:50.052701 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:50.053089 (+ 388us) server_negotiation.cc:299] Negotiation successful
0924 19:04:50.053230 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 19:04:50.053586 18767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dedce8eb54427d892af94b5b3829b3" candidate_uuid: "6f0e0296a4404693891b4746efd56c10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1cba60563724e79b1bd286b94752786" is_pre_election: true
I20260924 19:04:50.054177 18767 raft_consensus.cc:2454] T 67dedce8eb54427d892af94b5b3829b3 P f1cba60563724e79b1bd286b94752786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f0e0296a4404693891b4746efd56c10 in term 0.
I20260924 19:04:50.054567 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dedce8eb54427d892af94b5b3829b3" candidate_uuid: "6f0e0296a4404693891b4746efd56c10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57036440df74406c9d3aa5fa0fa8b436" is_pre_election: true
I20260924 19:04:50.054819 18969 leader_election.cc:302] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f0e0296a4404693891b4746efd56c10, f1cba60563724e79b1bd286b94752786; no voters:
I20260924 19:04:50.055071 18901 raft_consensus.cc:2454] T 67dedce8eb54427d892af94b5b3829b3 P 57036440df74406c9d3aa5fa0fa8b436 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f0e0296a4404693891b4746efd56c10 in term 0.
I20260924 19:04:50.055249 19104 raft_consensus.cc:2781] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:04:50.055397 19104 raft_consensus.cc:482] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:04:50.055509 19104 raft_consensus.cc:3037] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:04:50.057768 19104 raft_consensus.cc:504] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f0e0296a4404693891b4746efd56c10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41013 } } peers { permanent_uuid: "f1cba60563724e79b1bd286b94752786" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37933 } } peers { permanent_uuid: "57036440df74406c9d3aa5fa0fa8b436" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45525 } }
I20260924 19:04:50.058327 19104 leader_election.cc:288] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10 [CANDIDATE]: Term 1 election: Requested vote from peers f1cba60563724e79b1bd286b94752786 (127.0.97.1:37933), 57036440df74406c9d3aa5fa0fa8b436 (127.0.97.2:45525)
I20260924 19:04:50.058763 18767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dedce8eb54427d892af94b5b3829b3" candidate_uuid: "6f0e0296a4404693891b4746efd56c10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1cba60563724e79b1bd286b94752786"
I20260924 19:04:50.058821 18901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dedce8eb54427d892af94b5b3829b3" candidate_uuid: "6f0e0296a4404693891b4746efd56c10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57036440df74406c9d3aa5fa0fa8b436"
I20260924 19:04:50.058959 18767 raft_consensus.cc:3037] T 67dedce8eb54427d892af94b5b3829b3 P f1cba60563724e79b1bd286b94752786 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:04:50.059079 18901 raft_consensus.cc:3037] T 67dedce8eb54427d892af94b5b3829b3 P 57036440df74406c9d3aa5fa0fa8b436 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:04:50.061411 18767 raft_consensus.cc:2454] T 67dedce8eb54427d892af94b5b3829b3 P f1cba60563724e79b1bd286b94752786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f0e0296a4404693891b4746efd56c10 in term 1.
I20260924 19:04:50.061589 18901 raft_consensus.cc:2454] T 67dedce8eb54427d892af94b5b3829b3 P 57036440df74406c9d3aa5fa0fa8b436 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f0e0296a4404693891b4746efd56c10 in term 1.
I20260924 19:04:50.061934 18969 leader_election.cc:302] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f0e0296a4404693891b4746efd56c10, f1cba60563724e79b1bd286b94752786; no voters:
I20260924 19:04:50.062237 19104 raft_consensus.cc:2781] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:04:50.062661 19104 raft_consensus.cc:686] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10 [term 1 LEADER]: Becoming Leader. State: Replica: 6f0e0296a4404693891b4746efd56c10, State: Running, Role: LEADER
I20260924 19:04:50.063215 19104 consensus_queue.cc:237] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f0e0296a4404693891b4746efd56c10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41013 } } peers { permanent_uuid: "f1cba60563724e79b1bd286b94752786" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37933 } } peers { permanent_uuid: "57036440df74406c9d3aa5fa0fa8b436" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45525 } }
I20260924 19:04:50.067549 18623 catalog_manager.cc:5495] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f0e0296a4404693891b4746efd56c10 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "6f0e0296a4404693891b4746efd56c10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f0e0296a4404693891b4746efd56c10" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1cba60563724e79b1bd286b94752786" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57036440df74406c9d3aa5fa0fa8b436" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45525 } health_report { overall_health: UNKNOWN } } }
W20260924 19:04:50.085465 18813 tablet.cc:2369] T 67dedce8eb54427d892af94b5b3829b3 P f1cba60563724e79b1bd286b94752786: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:04:50.098320 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:50.092474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34906 (local address 127.0.97.3:41013)
0924 19:04:50.092695 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:50.092701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:50.092767 (+ 66us) server_negotiation.cc:407] Connection header received
0924 19:04:50.093010 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:50.093018 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:50.093154 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:50.093335 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:04:50.093810 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:50.094573 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:50.096000 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:50.096414 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:50.096557 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:50.097154 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:04:50.097731 (+ 577us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:04:50.097920 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:50.098037 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 19:04:50.098084 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 19:04:50.106052 18901 raft_consensus.cc:1261] T 67dedce8eb54427d892af94b5b3829b3 P 57036440df74406c9d3aa5fa0fa8b436 [term 1 FOLLOWER]: Refusing update from remote peer 6f0e0296a4404693891b4746efd56c10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:04:50.106092 18767 raft_consensus.cc:1261] T 67dedce8eb54427d892af94b5b3829b3 P f1cba60563724e79b1bd286b94752786 [term 1 FOLLOWER]: Refusing update from remote peer 6f0e0296a4404693891b4746efd56c10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:04:50.106959 19108 consensus_queue.cc:1035] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1cba60563724e79b1bd286b94752786" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:04:50.107296 19104 consensus_queue.cc:1035] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57036440df74406c9d3aa5fa0fa8b436" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:04:50.130641 18623 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:04:50.130910 18623 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:04:50.134567 18623 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43540fde65f640db982921ebf41941e4: Sending DeleteTablet for 3 replicas of tablet 67dedce8eb54427d892af94b5b3829b3
I20260924 19:04:50.135660 19015 tablet_service.cc:1514] Processing DeleteTablet for tablet 67dedce8eb54427d892af94b5b3829b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:04:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34892
I20260924 19:04:50.136159 18881 tablet_service.cc:1514] Processing DeleteTablet for tablet 67dedce8eb54427d892af94b5b3829b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:04:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42388
I20260924 19:04:50.136682 19120 tablet_replica.cc:330] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10: stopping tablet replica
I20260924 19:04:50.137072 18747 tablet_service.cc:1514] Processing DeleteTablet for tablet 67dedce8eb54427d892af94b5b3829b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:04:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53234
I20260924 19:04:50.137133 19120 raft_consensus.cc:2229] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:04:50.137903 19120 raft_consensus.cc:2258] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:04:50.137969 19122 tablet_replica.cc:330] T 67dedce8eb54427d892af94b5b3829b3 P f1cba60563724e79b1bd286b94752786: stopping tablet replica
I20260924 19:04:50.138374 19122 raft_consensus.cc:2229] T 67dedce8eb54427d892af94b5b3829b3 P f1cba60563724e79b1bd286b94752786 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:04:50.138850 19122 raft_consensus.cc:2258] T 67dedce8eb54427d892af94b5b3829b3 P f1cba60563724e79b1bd286b94752786 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:04:50.140024 19120 ts_tablet_manager.cc:1905] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:04:50.140633 19122 ts_tablet_manager.cc:1905] T 67dedce8eb54427d892af94b5b3829b3 P f1cba60563724e79b1bd286b94752786: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:04:50.141086 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 18684
I20260924 19:04:50.147513 19122 ts_tablet_manager.cc:1918] T 67dedce8eb54427d892af94b5b3829b3 P f1cba60563724e79b1bd286b94752786: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:04:50.147661 19122 log.cc:1196] T 67dedce8eb54427d892af94b5b3829b3 P f1cba60563724e79b1bd286b94752786: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-0/wal/wals/67dedce8eb54427d892af94b5b3829b3
I20260924 19:04:50.148205 19122 ts_tablet_manager.cc:1939] T 67dedce8eb54427d892af94b5b3829b3 P f1cba60563724e79b1bd286b94752786: Deleting consensus metadata
I20260924 19:04:50.149737 19120 ts_tablet_manager.cc:1918] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:04:50.149961 19120 log.cc:1196] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-2/wal/wals/67dedce8eb54427d892af94b5b3829b3
I20260924 19:04:50.150096 18608 catalog_manager.cc:4841] TS f1cba60563724e79b1bd286b94752786 (127.0.97.1:37933): tablet 67dedce8eb54427d892af94b5b3829b3 (table test-table [id=40a26d58527448b3bb2b106c5c176842]) successfully deleted
I20260924 19:04:50.150580 19120 ts_tablet_manager.cc:1939] T 67dedce8eb54427d892af94b5b3829b3 P 6f0e0296a4404693891b4746efd56c10: Deleting consensus metadata
I20260924 19:04:50.152460 18610 catalog_manager.cc:4841] TS 6f0e0296a4404693891b4746efd56c10 (127.0.97.3:41013): tablet 67dedce8eb54427d892af94b5b3829b3 (table test-table [id=40a26d58527448b3bb2b106c5c176842]) successfully deleted
I20260924 19:04:50.154704 19121 tablet_replica.cc:330] T 67dedce8eb54427d892af94b5b3829b3 P 57036440df74406c9d3aa5fa0fa8b436: stopping tablet replica
I20260924 19:04:50.155143 19121 raft_consensus.cc:2229] T 67dedce8eb54427d892af94b5b3829b3 P 57036440df74406c9d3aa5fa0fa8b436 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:04:50.155666 19121 raft_consensus.cc:2258] T 67dedce8eb54427d892af94b5b3829b3 P 57036440df74406c9d3aa5fa0fa8b436 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:04:50.157610 19121 ts_tablet_manager.cc:1905] T 67dedce8eb54427d892af94b5b3829b3 P 57036440df74406c9d3aa5fa0fa8b436: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 19:04:50 dist-test-slave-9fn1 krb5kdc[18578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 19:04:50.164675 19121 ts_tablet_manager.cc:1918] T 67dedce8eb54427d892af94b5b3829b3 P 57036440df74406c9d3aa5fa0fa8b436: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:04:50.164832 19121 log.cc:1196] T 67dedce8eb54427d892af94b5b3829b3 P 57036440df74406c9d3aa5fa0fa8b436: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276386554690-388-0/minicluster-data/ts-1/wal/wals/67dedce8eb54427d892af94b5b3829b3
I20260924 19:04:50.165373 19121 ts_tablet_manager.cc:1939] T 67dedce8eb54427d892af94b5b3829b3 P 57036440df74406c9d3aa5fa0fa8b436: Deleting consensus metadata
I20260924 19:04:50.167213 18608 catalog_manager.cc:4841] TS 57036440df74406c9d3aa5fa0fa8b436 (127.0.97.2:45525): tablet 67dedce8eb54427d892af94b5b3829b3 (table test-table [id=40a26d58527448b3bb2b106c5c176842]) successfully deleted
I20260924 19:04:50.169220 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:50.142884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53248 (local address 127.0.97.1:37933)
0924 19:04:50.143074 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:50.143081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:50.154056 (+ 10975us) server_negotiation.cc:407] Connection header received
0924 19:04:50.154239 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:50.154247 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:50.154360 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:50.154460 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:50.155944 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:50.156886 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:50.158385 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:50.158791 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:50.158948 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:50.163008 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:50.163020 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:50.163034 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:50.163065 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:50.166231 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:50.167215 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:50.167224 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:50.167227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:50.167381 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:50.167797 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:50.167800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:50.167802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:50.168139 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:50.168353 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:50.168717 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 19:04:50.168877 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 19:04:50.169966 18807 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:04:50.271503 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 18684
I20260924 19:04:50.296217 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 18818
Sep 24 19:04:50 dist-test-slave-9fn1 krb5kdc[18578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 19:04:50.313985 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:50.297099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42398 (local address 127.0.97.2:45525)
0924 19:04:50.297263 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:50.297267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:50.297285 (+ 18us) server_negotiation.cc:407] Connection header received
0924 19:04:50.297431 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:50.297435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:50.297545 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:50.297641 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:50.299290 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:50.300015 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:50.302017 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:50.302405 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:50.302497 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:50.307466 (+ 4969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:50.307486 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:50.307506 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:50.307547 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:50.311258 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:50.311894 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:50.311904 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:50.311908 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:50.312052 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:50.312405 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:50.312410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:50.312414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:50.312878 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:50.313198 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:50.313356 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 19:04:50.313595 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 19:04:50.314760 18941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:04:50.405375 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 18818
I20260924 19:04:50.420178 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 18952
Sep 24 19:04:50 dist-test-slave-9fn1 krb5kdc[18578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 19:04:50.438970 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:50.421188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34922 (local address 127.0.97.3:41013)
0924 19:04:50.421349 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:50.421353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:50.421371 (+ 18us) server_negotiation.cc:407] Connection header received
0924 19:04:50.421458 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:50.421462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:50.421583 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:50.421672 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:50.423509 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:50.424335 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:50.426442 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:50.426892 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:50.426986 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:50.432692 (+ 5706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:50.432705 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:50.432720 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:50.432757 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:50.436076 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:50.436797 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:50.436807 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:50.436809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:50.436962 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:50.437345 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:50.437351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:50.437354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:50.437801 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:50.438152 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:50.438370 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 19:04:50.438606 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 19:04:50.439694 19075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:04:50.560717 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 18952
I20260924 19:04:50.581260 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 18591
I20260924 19:04:50.582665 18653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:04:50.693955 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 18591
2026-09-24T19:04:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4251 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T19:04:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:04:50Z Disabled control of system clock
I20260924 19:04:50.768527 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36101
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:04:51.091298 19131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:51.091562 19131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:51.091642 19131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:51.098616 19131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:04:51.098753 19131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:51.098790 19131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:51.098838 19131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:04:51.098882 19131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:04:51.107625 19131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36101
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:51.108891 19131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:51.110615 19131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:51.122174 19137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:51.122118 19139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:51.122212 19136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:51.123173 19131 server_base.cc:1034] running on GCE node
I20260924 19:04:51.123984 19131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:51.125319 19131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:51.126536 19131 hybrid_clock.cc:648] HybridClock initialized: now 1790276691126478 us; error 59 us; skew 500 ppm
I20260924 19:04:51.126865 19131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:04:51.129066 19131 webserver.cc:460] Webserver started at http://127.0.97.62:34565/ using document root <none> and password file <none>
I20260924 19:04:51.129855 19131 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:51.129984 19131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:51.130295 19131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:04:51.133076 19131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "9ef597bd95bc4ac387f5794b2af82a6e"
format_stamp: "Formatted at 2026-09-24 19:04:51 on dist-test-slave-9fn1"
server_key: "31f4e47ce880b867b54ccd8fb1e5df85"
server_key_iv: "cdfbd43ed37e5578ccd62030b64ed19b"
server_key_version: "encryptionkey@0"
I20260924 19:04:51.133917 19131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "9ef597bd95bc4ac387f5794b2af82a6e"
format_stamp: "Formatted at 2026-09-24 19:04:51 on dist-test-slave-9fn1"
server_key: "31f4e47ce880b867b54ccd8fb1e5df85"
server_key_iv: "cdfbd43ed37e5578ccd62030b64ed19b"
server_key_version: "encryptionkey@0"
I20260924 19:04:51.138713 19131 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 19:04:51.141989 19145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:51.143327 19131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 19:04:51.143543 19131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "9ef597bd95bc4ac387f5794b2af82a6e"
format_stamp: "Formatted at 2026-09-24 19:04:51 on dist-test-slave-9fn1"
server_key: "31f4e47ce880b867b54ccd8fb1e5df85"
server_key_iv: "cdfbd43ed37e5578ccd62030b64ed19b"
server_key_version: "encryptionkey@0"
I20260924 19:04:51.143755 19131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:51.168803 19131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:51.169787 19131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:51.170082 19131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:51.188828 19131 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36101
I20260924 19:04:51.188859 19196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36101 every 8 connection(s)
I20260924 19:04:51.190250 19131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:04:51.193915 19197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:04:51.200172 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 19131
I20260924 19:04:51.200369 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:04:51.200845 388 external_mini_cluster.cc:1428] Setting key 1bdece56c2aa924d9f66e7a59bcff5af
I20260924 19:04:51.204761 19197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ef597bd95bc4ac387f5794b2af82a6e: Bootstrap starting.
I20260924 19:04:51.208225 19197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ef597bd95bc4ac387f5794b2af82a6e: Neither blocks nor log segments found. Creating new log.
I20260924 19:04:51.209431 19197 log.cc:824] T 00000000000000000000000000000000 P 9ef597bd95bc4ac387f5794b2af82a6e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:04:51.212493 19197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ef597bd95bc4ac387f5794b2af82a6e: No bootstrap required, opened a new log
I20260924 19:04:51.214990 19200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:51.203144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38620 (local address 127.0.97.62:36101)
0924 19:04:51.204099 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:51.204110 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:51.204150 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:04:51.205995 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:51.206017 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:51.206489 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:51.206834 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:51.207452 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:51.208823 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:51.210627 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:51.211295 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:51.211427 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:51.212710 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:51.212718 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:51.212732 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:04:51.213268 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:51.213353 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:04:51.213374 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:51.213457 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:51.213729 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 19:04:51.214032 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":343,"threads_started":1}
I20260924 19:04:51.220124 19197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ef597bd95bc4ac387f5794b2af82a6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef597bd95bc4ac387f5794b2af82a6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36101 } }
I20260924 19:04:51.220477 19197 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ef597bd95bc4ac387f5794b2af82a6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:04:51.220602 19197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ef597bd95bc4ac387f5794b2af82a6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef597bd95bc4ac387f5794b2af82a6e, State: Initialized, Role: FOLLOWER
I20260924 19:04:51.221128 19197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ef597bd95bc4ac387f5794b2af82a6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef597bd95bc4ac387f5794b2af82a6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36101 } }
I20260924 19:04:51.221379 19197 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ef597bd95bc4ac387f5794b2af82a6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:04:51.221535 19197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ef597bd95bc4ac387f5794b2af82a6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:04:51.221717 19197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ef597bd95bc4ac387f5794b2af82a6e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:04:51.223774 19197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ef597bd95bc4ac387f5794b2af82a6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef597bd95bc4ac387f5794b2af82a6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36101 } }
I20260924 19:04:51.224231 19197 leader_election.cc:302] T 00000000000000000000000000000000 P 9ef597bd95bc4ac387f5794b2af82a6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ef597bd95bc4ac387f5794b2af82a6e; no voters:
I20260924 19:04:51.224787 19197 leader_election.cc:288] T 00000000000000000000000000000000 P 9ef597bd95bc4ac387f5794b2af82a6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:04:51.224927 19202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ef597bd95bc4ac387f5794b2af82a6e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:04:51.225755 19202 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ef597bd95bc4ac387f5794b2af82a6e [term 1 LEADER]: Becoming Leader. State: Replica: 9ef597bd95bc4ac387f5794b2af82a6e, State: Running, Role: LEADER
I20260924 19:04:51.226348 19202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ef597bd95bc4ac387f5794b2af82a6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef597bd95bc4ac387f5794b2af82a6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36101 } }
I20260924 19:04:51.226871 19197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ef597bd95bc4ac387f5794b2af82a6e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:04:51.229239 19204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef597bd95bc4ac387f5794b2af82a6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ef597bd95bc4ac387f5794b2af82a6e. Latest consensus state: current_term: 1 leader_uuid: "9ef597bd95bc4ac387f5794b2af82a6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef597bd95bc4ac387f5794b2af82a6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36101 } } }
I20260924 19:04:51.229703 19204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef597bd95bc4ac387f5794b2af82a6e [sys.catalog]: This master's current role is: LEADER
I20260924 19:04:51.233680 19203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef597bd95bc4ac387f5794b2af82a6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ef597bd95bc4ac387f5794b2af82a6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef597bd95bc4ac387f5794b2af82a6e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36101 } } }
I20260924 19:04:51.234063 19203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef597bd95bc4ac387f5794b2af82a6e [sys.catalog]: This master's current role is: LEADER
I20260924 19:04:51.234794 19211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:04:51.240088 19211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:04:51.248615 19211 catalog_manager.cc:1347] Generated new cluster ID: 61f114fad5a74f6fa3640e3c526854c2
I20260924 19:04:51.248723 19211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:04:51.286473 19211 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:04:51.287833 19211 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:04:51.295289 19211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ef597bd95bc4ac387f5794b2af82a6e: Generated new TSK 0
I20260924 19:04:51.296133 19211 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:04:51.301148 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36101
--builtin_ntp_servers=127.0.97.20:38971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:04:51.632031 19221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:51.632328 19221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:51.632417 19221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:51.639756 19221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:51.639849 19221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:51.639951 19221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:04:51.649103 19221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:51.650295 19221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:51.651922 19221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:51.665114 19226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:51.665160 19229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:51.665184 19227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:51.665946 19221 server_base.cc:1034] running on GCE node
I20260924 19:04:51.666451 19221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:51.667289 19221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:51.668481 19221 hybrid_clock.cc:648] HybridClock initialized: now 1790276691668430 us; error 52 us; skew 500 ppm
I20260924 19:04:51.668753 19221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:04:51.670970 19221 webserver.cc:460] Webserver started at http://127.0.97.1:45859/ using document root <none> and password file <none>
I20260924 19:04:51.671583 19221 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:51.671666 19221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:51.671924 19221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:04:51.674718 19221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-0/data/instance:
uuid: "73d7b4d5e28a4c8e870b961996b1036d"
format_stamp: "Formatted at 2026-09-24 19:04:51 on dist-test-slave-9fn1"
server_key: "481dd62e3af331effda09baf1732b1bb"
server_key_iv: "d935572fbe9758ed3fe8096cb29463fe"
server_key_version: "encryptionkey@0"
I20260924 19:04:51.675760 19221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-0/wal/instance:
uuid: "73d7b4d5e28a4c8e870b961996b1036d"
format_stamp: "Formatted at 2026-09-24 19:04:51 on dist-test-slave-9fn1"
server_key: "481dd62e3af331effda09baf1732b1bb"
server_key_iv: "d935572fbe9758ed3fe8096cb29463fe"
server_key_version: "encryptionkey@0"
I20260924 19:04:51.680801 19221 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260924 19:04:51.684325 19235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:51.685742 19221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:04:51.685876 19221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "73d7b4d5e28a4c8e870b961996b1036d"
format_stamp: "Formatted at 2026-09-24 19:04:51 on dist-test-slave-9fn1"
server_key: "481dd62e3af331effda09baf1732b1bb"
server_key_iv: "d935572fbe9758ed3fe8096cb29463fe"
server_key_version: "encryptionkey@0"
I20260924 19:04:51.686057 19221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:51.702198 19221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:51.702971 19221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:51.703210 19221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:51.704083 19221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:04:51.705590 19221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:04:51.705688 19221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:51.705765 19221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:04:51.705822 19221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:51.739226 19221 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34235
I20260924 19:04:51.739277 19347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34235 every 8 connection(s)
I20260924 19:04:51.740504 19221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:04:51.748621 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 19221
I20260924 19:04:51.748834 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-0/wal/instance
I20260924 19:04:51.749426 388 external_mini_cluster.cc:1428] Setting key 6237fc0410d91bc5d78ab1853d189b91
I20260924 19:04:51.752275 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36101
--builtin_ntp_servers=127.0.97.20:38971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 19:04:51.753860 19350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:51.743023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35063 (local address 127.0.97.62:36101)
0924 19:04:51.743491 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:51.743497 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:51.744639 (+ 1142us) server_negotiation.cc:407] Connection header received
0924 19:04:51.746484 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:51.746488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:51.746560 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:51.746661 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:51.747783 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:51.748635 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:51.750769 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:51.751324 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:51.751450 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:51.752991 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:51.752996 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:51.752999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:04:51.753017 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:51.753058 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:04:51.753069 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:51.753166 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:51.753382 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 19:04:51.753475 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":345,"thread_start_us":231,"threads_started":1}
I20260924 19:04:51.754858 19351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:51.743618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36101 (local address 127.0.97.1:35063)
0924 19:04:51.744480 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:51.744514 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:51.746232 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:51.746895 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:51.746907 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:51.747011 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:04:51.747632 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:51.747659 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:51.748842 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:51.748850 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:51.750639 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:51.750653 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:51.751552 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:51.751560 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:51.751795 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:51.752671 (+ 876us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:04:51.752674 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:04:51.752712 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:04:51.752788 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:04:51.753165 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:04:51.753178 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:04:51.753188 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 19:04:51.753373 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 19:04:51.753816 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":272,"threads_started":1}
I20260924 19:04:51.756358 19348 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36101
I20260924 19:04:51.756661 19348 heartbeater.cc:464] Registering TS with master...
I20260924 19:04:51.757418 19348 heartbeater.cc:511] Master 127.0.97.62:36101 requested a full tablet report, sending...
I20260924 19:04:51.759570 19162 ts_manager.cc:194] Registered new tserver with Master: 73d7b4d5e28a4c8e870b961996b1036d (127.0.97.1:34235)
I20260924 19:04:51.761036 19162 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:35063
W20260924 19:04:52.075114 19352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:52.075363 19352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:52.075519 19352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:52.082517 19352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:52.082603 19352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:52.082795 19352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:04:52.091571 19352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:52.092715 19352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:52.094408 19352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:52.106690 19360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:52.106833 19357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:52.106830 19358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:52.107653 19352 server_base.cc:1034] running on GCE node
I20260924 19:04:52.108264 19352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:52.109130 19352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:52.110355 19352 hybrid_clock.cc:648] HybridClock initialized: now 1790276692110316 us; error 46 us; skew 500 ppm
I20260924 19:04:52.110697 19352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:04:52.112757 19352 webserver.cc:460] Webserver started at http://127.0.97.2:44515/ using document root <none> and password file <none>
I20260924 19:04:52.113399 19352 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:52.113551 19352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:52.113880 19352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:04:52.116616 19352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-1/data/instance:
uuid: "8334f164b8894d0abda437accbb2bdbb"
format_stamp: "Formatted at 2026-09-24 19:04:52 on dist-test-slave-9fn1"
server_key: "ddd5c5d210867cfb1801abc062c1268a"
server_key_iv: "fbc93827652dd75faf29bca5d104271e"
server_key_version: "encryptionkey@0"
I20260924 19:04:52.117344 19352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-1/wal/instance:
uuid: "8334f164b8894d0abda437accbb2bdbb"
format_stamp: "Formatted at 2026-09-24 19:04:52 on dist-test-slave-9fn1"
server_key: "ddd5c5d210867cfb1801abc062c1268a"
server_key_iv: "fbc93827652dd75faf29bca5d104271e"
server_key_version: "encryptionkey@0"
I20260924 19:04:52.122078 19352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 19:04:52.125267 19366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:52.126513 19352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:04:52.126688 19352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "8334f164b8894d0abda437accbb2bdbb"
format_stamp: "Formatted at 2026-09-24 19:04:52 on dist-test-slave-9fn1"
server_key: "ddd5c5d210867cfb1801abc062c1268a"
server_key_iv: "fbc93827652dd75faf29bca5d104271e"
server_key_version: "encryptionkey@0"
I20260924 19:04:52.126914 19352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:52.144979 19352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:52.145922 19352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:52.146205 19352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:52.147259 19352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:04:52.148794 19352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:04:52.148936 19352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:52.149060 19352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:04:52.149147 19352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:52.181690 19352 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46331
I20260924 19:04:52.181773 19478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46331 every 8 connection(s)
I20260924 19:04:52.182967 19352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:04:52.186380 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 19352
I20260924 19:04:52.186587 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-1/wal/instance
I20260924 19:04:52.187011 388 external_mini_cluster.cc:1428] Setting key f7ffeff83aac56d1322b81ea48eb0ca0
I20260924 19:04:52.189946 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36101
--builtin_ntp_servers=127.0.97.20:38971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 19:04:52.198032 19350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:52.185557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37707 (local address 127.0.97.62:36101)
0924 19:04:52.185752 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:52.185756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:52.187250 (+ 1494us) server_negotiation.cc:407] Connection header received
0924 19:04:52.189546 (+ 2296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:52.189554 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:52.189641 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:52.189756 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:52.191060 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.192272 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.194608 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.195344 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.195492 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:52.197065 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:52.197074 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:52.197082 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:04:52.197108 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:52.197152 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:04:52.197171 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:52.197270 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:52.197522 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 19:04:52.197662 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 19:04:52.199075 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:52.186019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36101 (local address 127.0.97.2:37707)
0924 19:04:52.187080 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:52.187131 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:52.189170 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:52.189992 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:52.190015 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:52.190129 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:04:52.190882 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:52.190903 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.192424 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.192430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:52.194450 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:52.194464 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.195572 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.195579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:52.195855 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:52.196706 (+ 851us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:04:52.196710 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:04:52.196760 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:04:52.196860 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:04:52.197273 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:04:52.197283 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:04:52.197291 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 19:04:52.197461 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 19:04:52.197869 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":361,"threads_started":1}
I20260924 19:04:52.200542 19479 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36101
I20260924 19:04:52.200848 19479 heartbeater.cc:464] Registering TS with master...
I20260924 19:04:52.201797 19479 heartbeater.cc:511] Master 127.0.97.62:36101 requested a full tablet report, sending...
I20260924 19:04:52.203312 19162 ts_manager.cc:194] Registered new tserver with Master: 8334f164b8894d0abda437accbb2bdbb (127.0.97.2:46331)
I20260924 19:04:52.204599 19162 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:37707
W20260924 19:04:52.510217 19482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:52.510478 19482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:52.510562 19482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:52.517603 19482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:52.517697 19482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:52.517800 19482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:04:52.526510 19482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:52.527639 19482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:52.529294 19482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:52.545238 19488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:52.545495 19487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:52.546590 19490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:52.547039 19482 server_base.cc:1034] running on GCE node
I20260924 19:04:52.547658 19482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:52.548606 19482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:52.549819 19482 hybrid_clock.cc:648] HybridClock initialized: now 1790276692549781 us; error 49 us; skew 500 ppm
I20260924 19:04:52.550113 19482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:04:52.552295 19482 webserver.cc:460] Webserver started at http://127.0.97.3:33977/ using document root <none> and password file <none>
I20260924 19:04:52.552949 19482 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:52.553074 19482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:52.553382 19482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:04:52.556227 19482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-2/data/instance:
uuid: "7dcfdbecd49849568726a961687b4a81"
format_stamp: "Formatted at 2026-09-24 19:04:52 on dist-test-slave-9fn1"
server_key: "fd1f71021042e2dee7a29577e1dd1a0f"
server_key_iv: "4ed811ec37656cc3a789e98ae0b8fe86"
server_key_version: "encryptionkey@0"
I20260924 19:04:52.556973 19482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7dcfdbecd49849568726a961687b4a81"
format_stamp: "Formatted at 2026-09-24 19:04:52 on dist-test-slave-9fn1"
server_key: "fd1f71021042e2dee7a29577e1dd1a0f"
server_key_iv: "4ed811ec37656cc3a789e98ae0b8fe86"
server_key_version: "encryptionkey@0"
I20260924 19:04:52.561947 19482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 19:04:52.565209 19496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:52.566479 19482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:04:52.566644 19482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "7dcfdbecd49849568726a961687b4a81"
format_stamp: "Formatted at 2026-09-24 19:04:52 on dist-test-slave-9fn1"
server_key: "fd1f71021042e2dee7a29577e1dd1a0f"
server_key_iv: "4ed811ec37656cc3a789e98ae0b8fe86"
server_key_version: "encryptionkey@0"
I20260924 19:04:52.566850 19482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:52.593072 19482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:52.593998 19482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:52.594276 19482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:52.595237 19482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:04:52.596735 19482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:04:52.596855 19482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:52.596968 19482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:04:52.597069 19482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:52.629587 19482 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34467
I20260924 19:04:52.629686 19608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34467 every 8 connection(s)
I20260924 19:04:52.630884 19482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:04:52.636654 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 19482
I20260924 19:04:52.636847 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-2/wal/instance
I20260924 19:04:52.637398 388 external_mini_cluster.cc:1428] Setting key d7355b283a68c8f4cd88bf5dcbf73025
I20260924 19:04:52.643517 19350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:52.633534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40765 (local address 127.0.97.62:36101)
0924 19:04:52.633699 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:52.633703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:52.635018 (+ 1315us) server_negotiation.cc:407] Connection header received
0924 19:04:52.636765 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:52.636772 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:52.636873 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:52.636983 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:52.638111 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.639203 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.640862 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.641301 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.641434 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:52.642691 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:52.642697 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:52.642702 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:04:52.642729 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:52.642760 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:04:52.642772 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:52.642868 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:52.643088 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 19:04:52.643206 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 19:04:52.644441 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:52.634030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36101 (local address 127.0.97.3:40765)
0924 19:04:52.634871 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:52.634928 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:52.636501 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:52.637195 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:52.637202 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:52.637292 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:04:52.637956 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:52.637974 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.639334 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.639339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:52.640717 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:52.640729 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.641543 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.641551 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:52.641783 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:52.642389 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:04:52.642391 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:04:52.642429 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:04:52.642501 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:04:52.642867 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:04:52.642880 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:04:52.642890 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 19:04:52.643065 (+ 175us) client_negotiation.cc:240] Negotiation successful
0924 19:04:52.643447 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":263,"threads_started":1}
I20260924 19:04:52.645709 19609 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36101
I20260924 19:04:52.645998 19609 heartbeater.cc:464] Registering TS with master...
I20260924 19:04:52.646809 19609 heartbeater.cc:511] Master 127.0.97.62:36101 requested a full tablet report, sending...
I20260924 19:04:52.648190 19162 ts_manager.cc:194] Registered new tserver with Master: 7dcfdbecd49849568726a961687b4a81 (127.0.97.3:34467)
I20260924 19:04:52.648968 19162 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:40765
I20260924 19:04:52.652333 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:04:52.661722 19350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:52.656235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38636 (local address 127.0.97.62:36101)
0924 19:04:52.656402 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:52.656407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:52.656880 (+ 473us) server_negotiation.cc:407] Connection header received
0924 19:04:52.657004 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:52.657007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:52.657068 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:52.657143 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:52.657559 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.658340 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.659637 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.660013 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.660099 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:52.661042 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:52.661046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:52.661050 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:04:52.661065 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:52.661093 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:04:52.661104 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:52.661219 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:52.661410 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 19:04:52.661450 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 19:04:52.665103 19162 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38636:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:04:52.671949 19162 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:04:52.705636 19621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:52.683334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34235 (local address 127.0.0.1:32808)
0924 19:04:52.695676 (+ 12342us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:52.695735 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:52.695947 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:52.697793 (+ 1846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:52.697800 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:52.697830 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:52.698322 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:52.698337 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.700542 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.700547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:52.703152 (+ 2605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:52.703167 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.704605 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.704611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:52.704824 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:52.704884 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 19:04:52.705021 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 19:04:52.705205 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12006,"mutex_wait_us":37,"thread_start_us":11816,"threads_started":1}
I20260924 19:04:52.705735 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:52.683355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34467 (local address 127.0.0.1:53454)
0924 19:04:52.697439 (+ 14084us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:52.697461 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:52.697617 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:52.698242 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:52.698248 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:52.698303 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:52.698729 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:52.698745 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.700417 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.700422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:52.703064 (+ 2642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:52.703078 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.704974 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.704982 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:52.705140 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:52.705169 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 19:04:52.705261 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 19:04:52.705374 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13882,"mutex_wait_us":45,"thread_start_us":14268,"threads_started":1}
I20260924 19:04:52.705927 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:52.683719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53454 (local address 127.0.97.3:34467)
0924 19:04:52.695161 (+ 11442us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:52.695173 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:52.697591 (+ 2418us) server_negotiation.cc:407] Connection header received
0924 19:04:52.697832 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:52.697839 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:52.698043 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:52.698248 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:52.698924 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.700240 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.703269 (+ 3029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.704654 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.704791 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:52.705048 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:52.705335 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 19:04:52.705529 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11249,"thread_start_us":11089,"threads_started":1}
I20260924 19:04:52.706802 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:52.696406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46758 (local address 127.0.97.2:46331)
0924 19:04:52.697048 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:52.697058 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:52.697818 (+ 760us) server_negotiation.cc:407] Connection header received
0924 19:04:52.698089 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:52.698101 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:52.698342 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:52.698516 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:52.699299 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.700790 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.703632 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.705010 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.705137 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:52.705322 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:52.706339 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0924 19:04:52.706495 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":342,"threads_started":1}
I20260924 19:04:52.706832 19623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:52.683429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32808 (local address 127.0.97.1:34235)
0924 19:04:52.697070 (+ 13641us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:52.697079 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:52.697115 (+ 36us) server_negotiation.cc:407] Connection header received
0924 19:04:52.697200 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:52.697205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:52.697504 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:52.698578 (+ 1074us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:52.698813 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.700387 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.703384 (+ 2997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.704407 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.704556 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:52.704833 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:52.706318 (+ 1485us) server_negotiation.cc:299] Negotiation successful
0924 19:04:52.706490 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13496,"thread_start_us":13390,"threads_started":1}
I20260924 19:04:52.707777 19626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:52.697065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46331 (local address 127.0.0.1:46758)
0924 19:04:52.697728 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:52.697744 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:52.697913 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:52.698586 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:52.698592 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:52.698627 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:52.699091 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:52.699104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.701092 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.701098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:52.703424 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:52.703439 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.705455 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.705462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:52.705691 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:52.706157 (+ 466us) client_negotiation.cc:769] Sending connection context
0924 19:04:52.707328 (+ 1171us) client_negotiation.cc:240] Negotiation successful
0924 19:04:52.707452 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":96,"thread_start_us":298,"threads_started":1}
I20260924 19:04:52.710078 19544 tablet_service.cc:1467] Processing CreateTablet for tablet 369dff5a13734a378d7b28d34137e2c2 (DEFAULT_TABLE table=test-table [id=21b46880eba7429380546f9173c737f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:04:52.710381 19283 tablet_service.cc:1467] Processing CreateTablet for tablet 369dff5a13734a378d7b28d34137e2c2 (DEFAULT_TABLE table=test-table [id=21b46880eba7429380546f9173c737f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:04:52.710415 19414 tablet_service.cc:1467] Processing CreateTablet for tablet 369dff5a13734a378d7b28d34137e2c2 (DEFAULT_TABLE table=test-table [id=21b46880eba7429380546f9173c737f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:04:52.711524 19414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 369dff5a13734a378d7b28d34137e2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:04:52.711576 19283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 369dff5a13734a378d7b28d34137e2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:04:52.711576 19544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 369dff5a13734a378d7b28d34137e2c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:04:52.723691 19627 tablet_bootstrap.cc:492] T 369dff5a13734a378d7b28d34137e2c2 P 8334f164b8894d0abda437accbb2bdbb: Bootstrap starting.
I20260924 19:04:52.725663 19628 tablet_bootstrap.cc:492] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d: Bootstrap starting.
I20260924 19:04:52.727051 19627 tablet_bootstrap.cc:654] T 369dff5a13734a378d7b28d34137e2c2 P 8334f164b8894d0abda437accbb2bdbb: Neither blocks nor log segments found. Creating new log.
I20260924 19:04:52.728334 19627 log.cc:824] T 369dff5a13734a378d7b28d34137e2c2 P 8334f164b8894d0abda437accbb2bdbb: Log is configured to *not* fsync() on all Append() calls
I20260924 19:04:52.729283 19628 tablet_bootstrap.cc:654] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d: Neither blocks nor log segments found. Creating new log.
I20260924 19:04:52.729785 19629 tablet_bootstrap.cc:492] T 369dff5a13734a378d7b28d34137e2c2 P 7dcfdbecd49849568726a961687b4a81: Bootstrap starting.
I20260924 19:04:52.730696 19628 log.cc:824] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:04:52.731552 19627 tablet_bootstrap.cc:492] T 369dff5a13734a378d7b28d34137e2c2 P 8334f164b8894d0abda437accbb2bdbb: No bootstrap required, opened a new log
I20260924 19:04:52.731845 19627 ts_tablet_manager.cc:1397] T 369dff5a13734a378d7b28d34137e2c2 P 8334f164b8894d0abda437accbb2bdbb: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 19:04:52.735256 19629 tablet_bootstrap.cc:654] T 369dff5a13734a378d7b28d34137e2c2 P 7dcfdbecd49849568726a961687b4a81: Neither blocks nor log segments found. Creating new log.
I20260924 19:04:52.736694 19629 log.cc:824] T 369dff5a13734a378d7b28d34137e2c2 P 7dcfdbecd49849568726a961687b4a81: Log is configured to *not* fsync() on all Append() calls
I20260924 19:04:52.739841 19629 tablet_bootstrap.cc:492] T 369dff5a13734a378d7b28d34137e2c2 P 7dcfdbecd49849568726a961687b4a81: No bootstrap required, opened a new log
I20260924 19:04:52.739867 19628 tablet_bootstrap.cc:492] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d: No bootstrap required, opened a new log
I20260924 19:04:52.740163 19628 ts_tablet_manager.cc:1397] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20260924 19:04:52.740156 19629 ts_tablet_manager.cc:1397] T 369dff5a13734a378d7b28d34137e2c2 P 7dcfdbecd49849568726a961687b4a81: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260924 19:04:52.740558 19627 raft_consensus.cc:348] T 369dff5a13734a378d7b28d34137e2c2 P 8334f164b8894d0abda437accbb2bdbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d7b4d5e28a4c8e870b961996b1036d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34235 } } peers { permanent_uuid: "7dcfdbecd49849568726a961687b4a81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34467 } } peers { permanent_uuid: "8334f164b8894d0abda437accbb2bdbb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46331 } }
I20260924 19:04:52.741005 19627 raft_consensus.cc:374] T 369dff5a13734a378d7b28d34137e2c2 P 8334f164b8894d0abda437accbb2bdbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:04:52.741137 19627 raft_consensus.cc:729] T 369dff5a13734a378d7b28d34137e2c2 P 8334f164b8894d0abda437accbb2bdbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8334f164b8894d0abda437accbb2bdbb, State: Initialized, Role: FOLLOWER
I20260924 19:04:52.741797 19627 consensus_queue.cc:260] T 369dff5a13734a378d7b28d34137e2c2 P 8334f164b8894d0abda437accbb2bdbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d7b4d5e28a4c8e870b961996b1036d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34235 } } peers { permanent_uuid: "7dcfdbecd49849568726a961687b4a81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34467 } } peers { permanent_uuid: "8334f164b8894d0abda437accbb2bdbb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46331 } }
I20260924 19:04:52.750769 19629 raft_consensus.cc:348] T 369dff5a13734a378d7b28d34137e2c2 P 7dcfdbecd49849568726a961687b4a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d7b4d5e28a4c8e870b961996b1036d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34235 } } peers { permanent_uuid: "7dcfdbecd49849568726a961687b4a81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34467 } } peers { permanent_uuid: "8334f164b8894d0abda437accbb2bdbb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46331 } }
I20260924 19:04:52.751327 19629 raft_consensus.cc:374] T 369dff5a13734a378d7b28d34137e2c2 P 7dcfdbecd49849568726a961687b4a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:04:52.751472 19629 raft_consensus.cc:729] T 369dff5a13734a378d7b28d34137e2c2 P 7dcfdbecd49849568726a961687b4a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dcfdbecd49849568726a961687b4a81, State: Initialized, Role: FOLLOWER
I20260924 19:04:52.752892 19627 ts_tablet_manager.cc:1428] T 369dff5a13734a378d7b28d34137e2c2 P 8334f164b8894d0abda437accbb2bdbb: Time spent starting tablet: real 0.021s user 0.016s sys 0.005s
I20260924 19:04:52.752828 19629 consensus_queue.cc:260] T 369dff5a13734a378d7b28d34137e2c2 P 7dcfdbecd49849568726a961687b4a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d7b4d5e28a4c8e870b961996b1036d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34235 } } peers { permanent_uuid: "7dcfdbecd49849568726a961687b4a81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34467 } } peers { permanent_uuid: "8334f164b8894d0abda437accbb2bdbb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46331 } }
I20260924 19:04:52.752831 19628 raft_consensus.cc:348] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d7b4d5e28a4c8e870b961996b1036d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34235 } } peers { permanent_uuid: "7dcfdbecd49849568726a961687b4a81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34467 } } peers { permanent_uuid: "8334f164b8894d0abda437accbb2bdbb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46331 } }
I20260924 19:04:52.753306 19628 raft_consensus.cc:374] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:04:52.753383 19479 heartbeater.cc:503] Master 127.0.97.62:36101 was elected leader, sending a full tablet report...
I20260924 19:04:52.753480 19628 raft_consensus.cc:729] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73d7b4d5e28a4c8e870b961996b1036d, State: Initialized, Role: FOLLOWER
I20260924 19:04:52.754508 19628 consensus_queue.cc:260] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d7b4d5e28a4c8e870b961996b1036d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34235 } } peers { permanent_uuid: "7dcfdbecd49849568726a961687b4a81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34467 } } peers { permanent_uuid: "8334f164b8894d0abda437accbb2bdbb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46331 } }
I20260924 19:04:52.759577 19348 heartbeater.cc:503] Master 127.0.97.62:36101 was elected leader, sending a full tablet report...
I20260924 19:04:52.760856 19628 ts_tablet_manager.cc:1428] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d: Time spent starting tablet: real 0.021s user 0.013s sys 0.004s
I20260924 19:04:52.761839 19609 heartbeater.cc:503] Master 127.0.97.62:36101 was elected leader, sending a full tablet report...
I20260924 19:04:52.762522 19629 ts_tablet_manager.cc:1428] T 369dff5a13734a378d7b28d34137e2c2 P 7dcfdbecd49849568726a961687b4a81: Time spent starting tablet: real 0.022s user 0.004s sys 0.017s
I20260924 19:04:52.835914 19635 raft_consensus.cc:482] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:04:52.836318 19635 raft_consensus.cc:504] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d7b4d5e28a4c8e870b961996b1036d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34235 } } peers { permanent_uuid: "7dcfdbecd49849568726a961687b4a81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34467 } } peers { permanent_uuid: "8334f164b8894d0abda437accbb2bdbb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46331 } }
I20260924 19:04:52.838419 19635 leader_election.cc:288] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7dcfdbecd49849568726a961687b4a81 (127.0.97.3:34467), 8334f164b8894d0abda437accbb2bdbb (127.0.97.2:46331)
I20260924 19:04:52.847874 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:52.838976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41649 (local address 127.0.97.3:34467)
0924 19:04:52.839276 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:52.839285 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:52.840221 (+ 936us) server_negotiation.cc:407] Connection header received
0924 19:04:52.840470 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:52.840476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:52.840553 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:52.840659 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:52.841279 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.842480 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.845372 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.846791 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.846997 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:52.847191 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:52.847450 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 19:04:52.847591 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260924 19:04:52.847874 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:52.839452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54555 (local address 127.0.97.2:46331)
0924 19:04:52.839720 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:52.839729 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:52.840463 (+ 734us) server_negotiation.cc:407] Connection header received
0924 19:04:52.840666 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:52.840670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:52.840737 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:52.840830 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:52.841627 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.842721 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.845372 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.846786 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.846950 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:52.847052 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:52.847450 (+ 398us) server_negotiation.cc:299] Negotiation successful
0924 19:04:52.847592 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 19:04:52.847972 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:52.838658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34467 (local address 127.0.97.1:41649)
0924 19:04:52.840085 (+ 1427us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:52.840107 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:52.840307 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:52.840690 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:52.840695 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:52.840722 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:52.841112 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:52.841125 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.842607 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.842611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:52.845130 (+ 2519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:52.845141 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.847044 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.847049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:52.847197 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:52.847301 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 19:04:52.847404 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 19:04:52.847538 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1261,"mutex_wait_us":198,"thread_start_us":1039,"threads_started":1}
I20260924 19:04:52.847994 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:52.838834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46331 (local address 127.0.97.1:54555)
0924 19:04:52.840355 (+ 1521us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:52.840369 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:52.840514 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:52.840870 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:52.840878 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:52.840975 (+ 97us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:52.841386 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:52.841402 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.842855 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.842860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:52.845126 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:52.845141 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.847005 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.847013 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:52.847299 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:52.847323 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 19:04:52.847464 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 19:04:52.847560 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1393,"mutex_wait_us":122,"spinlock_wait_cycles":30592,"thread_start_us":1223,"threads_started":1}
I20260924 19:04:52.848976 19434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "369dff5a13734a378d7b28d34137e2c2" candidate_uuid: "73d7b4d5e28a4c8e870b961996b1036d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8334f164b8894d0abda437accbb2bdbb" is_pre_election: true
I20260924 19:04:52.849160 19564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "369dff5a13734a378d7b28d34137e2c2" candidate_uuid: "73d7b4d5e28a4c8e870b961996b1036d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dcfdbecd49849568726a961687b4a81" is_pre_election: true
I20260924 19:04:52.849514 19434 raft_consensus.cc:2454] T 369dff5a13734a378d7b28d34137e2c2 P 8334f164b8894d0abda437accbb2bdbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73d7b4d5e28a4c8e870b961996b1036d in term 0.
I20260924 19:04:52.849772 19564 raft_consensus.cc:2454] T 369dff5a13734a378d7b28d34137e2c2 P 7dcfdbecd49849568726a961687b4a81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73d7b4d5e28a4c8e870b961996b1036d in term 0.
I20260924 19:04:52.850274 19238 leader_election.cc:302] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73d7b4d5e28a4c8e870b961996b1036d, 8334f164b8894d0abda437accbb2bdbb; no voters:
I20260924 19:04:52.850708 19635 raft_consensus.cc:2781] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:04:52.850838 19635 raft_consensus.cc:482] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:04:52.850914 19635 raft_consensus.cc:3037] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:04:52.853259 19635 raft_consensus.cc:504] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d7b4d5e28a4c8e870b961996b1036d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34235 } } peers { permanent_uuid: "7dcfdbecd49849568726a961687b4a81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34467 } } peers { permanent_uuid: "8334f164b8894d0abda437accbb2bdbb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46331 } }
I20260924 19:04:52.854060 19635 leader_election.cc:288] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d [CANDIDATE]: Term 1 election: Requested vote from peers 7dcfdbecd49849568726a961687b4a81 (127.0.97.3:34467), 8334f164b8894d0abda437accbb2bdbb (127.0.97.2:46331)
I20260924 19:04:52.854550 19564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "369dff5a13734a378d7b28d34137e2c2" candidate_uuid: "73d7b4d5e28a4c8e870b961996b1036d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dcfdbecd49849568726a961687b4a81"
I20260924 19:04:52.854595 19434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "369dff5a13734a378d7b28d34137e2c2" candidate_uuid: "73d7b4d5e28a4c8e870b961996b1036d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8334f164b8894d0abda437accbb2bdbb"
I20260924 19:04:52.854812 19434 raft_consensus.cc:3037] T 369dff5a13734a378d7b28d34137e2c2 P 8334f164b8894d0abda437accbb2bdbb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:04:52.854833 19564 raft_consensus.cc:3037] T 369dff5a13734a378d7b28d34137e2c2 P 7dcfdbecd49849568726a961687b4a81 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:04:52.858453 19434 raft_consensus.cc:2454] T 369dff5a13734a378d7b28d34137e2c2 P 8334f164b8894d0abda437accbb2bdbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73d7b4d5e28a4c8e870b961996b1036d in term 1.
I20260924 19:04:52.858453 19564 raft_consensus.cc:2454] T 369dff5a13734a378d7b28d34137e2c2 P 7dcfdbecd49849568726a961687b4a81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73d7b4d5e28a4c8e870b961996b1036d in term 1.
I20260924 19:04:52.858913 19238 leader_election.cc:302] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73d7b4d5e28a4c8e870b961996b1036d, 8334f164b8894d0abda437accbb2bdbb; no voters:
I20260924 19:04:52.859217 19635 raft_consensus.cc:2781] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:04:52.859694 19635 raft_consensus.cc:686] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d [term 1 LEADER]: Becoming Leader. State: Replica: 73d7b4d5e28a4c8e870b961996b1036d, State: Running, Role: LEADER
I20260924 19:04:52.860244 19635 consensus_queue.cc:237] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d7b4d5e28a4c8e870b961996b1036d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34235 } } peers { permanent_uuid: "7dcfdbecd49849568726a961687b4a81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34467 } } peers { permanent_uuid: "8334f164b8894d0abda437accbb2bdbb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46331 } }
I20260924 19:04:52.864651 19160 catalog_manager.cc:5495] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d reported cstate change: term changed from 0 to 1, leader changed from <none> to 73d7b4d5e28a4c8e870b961996b1036d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "73d7b4d5e28a4c8e870b961996b1036d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73d7b4d5e28a4c8e870b961996b1036d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7dcfdbecd49849568726a961687b4a81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8334f164b8894d0abda437accbb2bdbb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46331 } health_report { overall_health: UNKNOWN } } }
W20260924 19:04:52.882751 19610 tablet.cc:2369] T 369dff5a13734a378d7b28d34137e2c2 P 7dcfdbecd49849568726a961687b4a81: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:04:52.889128 19623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:52.882860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32810 (local address 127.0.97.1:34235)
0924 19:04:52.883069 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:52.883076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:52.883275 (+ 199us) server_negotiation.cc:407] Connection header received
0924 19:04:52.883509 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:52.883514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:52.883606 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:52.883771 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:04:52.884140 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.884840 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.886421 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.886885 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.887035 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:52.887875 (+ 840us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:04:52.888402 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:04:52.888616 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:52.888724 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 19:04:52.888772 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 19:04:52.897202 19434 raft_consensus.cc:1261] T 369dff5a13734a378d7b28d34137e2c2 P 8334f164b8894d0abda437accbb2bdbb [term 1 FOLLOWER]: Refusing update from remote peer 73d7b4d5e28a4c8e870b961996b1036d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:04:52.897202 19564 raft_consensus.cc:1261] T 369dff5a13734a378d7b28d34137e2c2 P 7dcfdbecd49849568726a961687b4a81 [term 1 FOLLOWER]: Refusing update from remote peer 73d7b4d5e28a4c8e870b961996b1036d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:04:52.898252 19638 consensus_queue.cc:1035] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8334f164b8894d0abda437accbb2bdbb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:04:52.898608 19635 consensus_queue.cc:1035] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7dcfdbecd49849568726a961687b4a81" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:04:52.910048 19641 mvcc.cc:205] Tried to move back new op lower bound from 7332973334097281024 to 7332973333959184384. Current Snapshot: MvccSnapshot[applied={T|T < 7332973334097281024}]
I20260924 19:04:52.911478 19642 mvcc.cc:205] Tried to move back new op lower bound from 7332973334097281024 to 7332973333959184384. Current Snapshot: MvccSnapshot[applied={T|T < 7332973334097281024}]
I20260924 19:04:52.925150 19160 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:04:52.925422 19160 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:04:52.928833 19160 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ef597bd95bc4ac387f5794b2af82a6e: Sending DeleteTablet for 3 replicas of tablet 369dff5a13734a378d7b28d34137e2c2
I20260924 19:04:52.930022 19414 tablet_service.cc:1514] Processing DeleteTablet for tablet 369dff5a13734a378d7b28d34137e2c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:04:52 UTC) from {username='slave'} at 127.0.0.1:46758
I20260924 19:04:52.930004 19283 tablet_service.cc:1514] Processing DeleteTablet for tablet 369dff5a13734a378d7b28d34137e2c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:04:52 UTC) from {username='slave'} at 127.0.0.1:32808
I20260924 19:04:52.930019 19544 tablet_service.cc:1514] Processing DeleteTablet for tablet 369dff5a13734a378d7b28d34137e2c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:04:52 UTC) from {username='slave'} at 127.0.0.1:53454
I20260924 19:04:52.934674 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 19221
I20260924 19:04:52.942273 19653 tablet_replica.cc:330] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d: stopping tablet replica
I20260924 19:04:52.942864 19653 raft_consensus.cc:2229] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:04:52.943706 19653 raft_consensus.cc:2258] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:04:52.947616 19653 ts_tablet_manager.cc:1905] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:04:52.955431 19653 ts_tablet_manager.cc:1918] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:04:52.955736 19653 log.cc:1196] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-0/wal/wals/369dff5a13734a378d7b28d34137e2c2
I20260924 19:04:52.956416 19653 ts_tablet_manager.cc:1939] T 369dff5a13734a378d7b28d34137e2c2 P 73d7b4d5e28a4c8e870b961996b1036d: Deleting consensus metadata
I20260924 19:04:52.958464 19148 catalog_manager.cc:4841] TS 73d7b4d5e28a4c8e870b961996b1036d (127.0.97.1:34235): tablet 369dff5a13734a378d7b28d34137e2c2 (table test-table [id=21b46880eba7429380546f9173c737f4]) successfully deleted
I20260924 19:04:52.960168 19652 tablet_replica.cc:330] T 369dff5a13734a378d7b28d34137e2c2 P 7dcfdbecd49849568726a961687b4a81: stopping tablet replica
I20260924 19:04:52.960637 19652 raft_consensus.cc:2229] T 369dff5a13734a378d7b28d34137e2c2 P 7dcfdbecd49849568726a961687b4a81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:04:52.961160 19652 raft_consensus.cc:2258] T 369dff5a13734a378d7b28d34137e2c2 P 7dcfdbecd49849568726a961687b4a81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:04:52.961992 19651 tablet_replica.cc:330] T 369dff5a13734a378d7b28d34137e2c2 P 8334f164b8894d0abda437accbb2bdbb: stopping tablet replica
I20260924 19:04:52.962832 19651 raft_consensus.cc:2229] T 369dff5a13734a378d7b28d34137e2c2 P 8334f164b8894d0abda437accbb2bdbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:04:52.963306 19651 raft_consensus.cc:2258] T 369dff5a13734a378d7b28d34137e2c2 P 8334f164b8894d0abda437accbb2bdbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:04:52.965397 19651 ts_tablet_manager.cc:1905] T 369dff5a13734a378d7b28d34137e2c2 P 8334f164b8894d0abda437accbb2bdbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:04:52.965452 19652 ts_tablet_manager.cc:1905] T 369dff5a13734a378d7b28d34137e2c2 P 7dcfdbecd49849568726a961687b4a81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:04:52.970113 19623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:52.935601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32818 (local address 127.0.97.1:34235)
0924 19:04:52.936447 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:52.936455 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:52.955111 (+ 18656us) server_negotiation.cc:407] Connection header received
0924 19:04:52.955734 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:52.955744 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:52.955901 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:52.955998 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:52.956961 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.958658 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.960658 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:52.961297 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:52.961441 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:52.962883 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:52.962894 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:52.962915 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:04:52.963734 (+ 819us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:52.963794 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:04:52.963819 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:52.963914 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:52.969587 (+ 5673us) server_negotiation.cc:299] Negotiation successful
0924 19:04:52.969701 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680}
I20260924 19:04:52.970206 19651 ts_tablet_manager.cc:1918] T 369dff5a13734a378d7b28d34137e2c2 P 8334f164b8894d0abda437accbb2bdbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:04:52.970345 19651 log.cc:1196] T 369dff5a13734a378d7b28d34137e2c2 P 8334f164b8894d0abda437accbb2bdbb: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-1/wal/wals/369dff5a13734a378d7b28d34137e2c2
I20260924 19:04:52.970861 19651 ts_tablet_manager.cc:1939] T 369dff5a13734a378d7b28d34137e2c2 P 8334f164b8894d0abda437accbb2bdbb: Deleting consensus metadata
I20260924 19:04:52.971633 19343 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:04:52.972128 19652 ts_tablet_manager.cc:1918] T 369dff5a13734a378d7b28d34137e2c2 P 7dcfdbecd49849568726a961687b4a81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:04:52.972292 19652 log.cc:1196] T 369dff5a13734a378d7b28d34137e2c2 P 7dcfdbecd49849568726a961687b4a81: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276386554690-388-0/minicluster-data/ts-2/wal/wals/369dff5a13734a378d7b28d34137e2c2
I20260924 19:04:52.972784 19652 ts_tablet_manager.cc:1939] T 369dff5a13734a378d7b28d34137e2c2 P 7dcfdbecd49849568726a961687b4a81: Deleting consensus metadata
I20260924 19:04:52.973702 19148 catalog_manager.cc:4841] TS 8334f164b8894d0abda437accbb2bdbb (127.0.97.2:46331): tablet 369dff5a13734a378d7b28d34137e2c2 (table test-table [id=21b46880eba7429380546f9173c737f4]) successfully deleted
I20260924 19:04:52.974436 19146 catalog_manager.cc:4841] TS 7dcfdbecd49849568726a961687b4a81 (127.0.97.3:34467): tablet 369dff5a13734a378d7b28d34137e2c2 (table test-table [id=21b46880eba7429380546f9173c737f4]) successfully deleted
I20260924 19:04:53.070497 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 19221
I20260924 19:04:53.089294 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 19352
I20260924 19:04:53.095674 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:53.090072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46762 (local address 127.0.97.2:46331)
0924 19:04:53.090248 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:53.090252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:53.090270 (+ 18us) server_negotiation.cc:407] Connection header received
0924 19:04:53.090402 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:53.090406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:53.090489 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:53.090573 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:53.090923 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:53.091639 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:53.092903 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:53.093288 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:53.093405 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:53.094472 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:53.094481 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:53.094495 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:04:53.095060 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:53.095106 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:04:53.095130 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:53.095215 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:53.095339 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 19:04:53.095422 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 19:04:53.096304 19474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:04:53.184770 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 19352
I20260924 19:04:53.201474 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 19482
I20260924 19:04:53.207865 19624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:53.202314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53464 (local address 127.0.97.3:34467)
0924 19:04:53.202482 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:53.202487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:53.202505 (+ 18us) server_negotiation.cc:407] Connection header received
0924 19:04:53.202600 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:53.202604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:53.202686 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:53.202778 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:53.203141 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:53.203895 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:53.205131 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:53.205541 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:53.205645 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:53.206605 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:53.206613 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:53.206628 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:04:53.207254 (+ 626us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:53.207316 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:04:53.207343 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:53.207460 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:53.207573 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 19:04:53.207654 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 19:04:53.208467 19604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:04:53.294552 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 19482
I20260924 19:04:53.310951 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 19131
I20260924 19:04:53.312029 19192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:04:53.420076 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 19131
2026-09-24T19:04:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2724 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:04:53 dist-test-slave-9fn1 krb5kdc[19661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:04:53 dist-test-slave-9fn1 krb5kdc[19661](info): set up 2 sockets
Sep 24 19:04:53 dist-test-slave-9fn1 krb5kdc[19661](info): commencing operation
krb5kdc: starting...
W20260924 19:04:54.582737 388 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:04:54 dist-test-slave-9fn1 krb5kdc[19661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:04:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:04:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 19:04:54.821424 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37707
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:04:55.141916 19674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:55.142234 19674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:55.142400 19674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:55.149472 19674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:04:55.149649 19674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:55.149760 19674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:55.149861 19674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:04:55.149941 19674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:04:55.159880 19674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:55.161221 19674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:55.162592 19674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:55.174170 19679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:55.174218 19682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:55.174247 19680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:55.175601 19674 server_base.cc:1034] running on GCE node
I20260924 19:04:55.176374 19674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:55.177836 19674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:55.179041 19674 hybrid_clock.cc:648] HybridClock initialized: now 1790276695178994 us; error 33 us; skew 500 ppm
I20260924 19:04:55.179416 19674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:04:55 dist-test-slave-9fn1 krb5kdc[19661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:04:55.189455 19674 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:04:55.191090 19674 webserver.cc:460] Webserver started at http://127.0.97.62:46245/ using document root <none> and password file <none>
I20260924 19:04:55.191812 19674 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:55.191946 19674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:55.192276 19674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:04:55.195313 19674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "9e7c26f3205741ad8e554380743a4eca"
format_stamp: "Formatted at 2026-09-24 19:04:55 on dist-test-slave-9fn1"
server_key: "a2adcfdad027e27b756f12cc565356bc"
server_key_iv: "0b1327a292a9cc195ce02a0fbbf99655"
server_key_version: "encryptionkey@0"
I20260924 19:04:55.196062 19674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "9e7c26f3205741ad8e554380743a4eca"
format_stamp: "Formatted at 2026-09-24 19:04:55 on dist-test-slave-9fn1"
server_key: "a2adcfdad027e27b756f12cc565356bc"
server_key_iv: "0b1327a292a9cc195ce02a0fbbf99655"
server_key_version: "encryptionkey@0"
I20260924 19:04:55.201263 19674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 19:04:55.204703 19689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:55.206102 19674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:04:55.206280 19674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "9e7c26f3205741ad8e554380743a4eca"
format_stamp: "Formatted at 2026-09-24 19:04:55 on dist-test-slave-9fn1"
server_key: "a2adcfdad027e27b756f12cc565356bc"
server_key_iv: "0b1327a292a9cc195ce02a0fbbf99655"
server_key_version: "encryptionkey@0"
I20260924 19:04:55.206471 19674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:55.234721 19674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:55.239051 19674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:55.239315 19674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:55.257433 19674 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37707
I20260924 19:04:55.257453 19740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37707 every 8 connection(s)
I20260924 19:04:55.258786 19674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:04:55.262411 19741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:04:55.265280 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 19674
I20260924 19:04:55.265461 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:04:55.266031 388 external_mini_cluster.cc:1428] Setting key 8887e5f0fa0dc8515f4538e67c797c96
I20260924 19:04:55.274681 19741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e7c26f3205741ad8e554380743a4eca: Bootstrap starting.
I20260924 19:04:55.278456 19741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e7c26f3205741ad8e554380743a4eca: Neither blocks nor log segments found. Creating new log.
I20260924 19:04:55.279785 19741 log.cc:824] T 00000000000000000000000000000000 P 9e7c26f3205741ad8e554380743a4eca: Log is configured to *not* fsync() on all Append() calls
Sep 24 19:04:55 dist-test-slave-9fn1 krb5kdc[19661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:04:55.282728 19741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e7c26f3205741ad8e554380743a4eca: No bootstrap required, opened a new log
I20260924 19:04:55.290611 19744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:55.267883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57880 (local address 127.0.97.62:37707)
0924 19:04:55.268766 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:55.268777 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:55.268816 (+ 39us) server_negotiation.cc:407] Connection header received
0924 19:04:55.270534 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:55.270555 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:55.270934 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:55.271250 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:55.272974 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:55.274311 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:55.276179 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:55.276840 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:55.276963 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:55.282116 (+ 5153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:55.282123 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:55.282136 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:55.282175 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:55.286149 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:55.286821 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:55.286829 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:55.286845 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:55.287048 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:55.287382 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:55.287389 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:55.287397 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:55.288107 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:55.288414 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:55.288745 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 19:04:55.289180 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":501,"threads_started":1}
I20260924 19:04:55.293483 19741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e7c26f3205741ad8e554380743a4eca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7c26f3205741ad8e554380743a4eca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37707 } }
I20260924 19:04:55.293871 19741 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e7c26f3205741ad8e554380743a4eca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:04:55.293955 19741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e7c26f3205741ad8e554380743a4eca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e7c26f3205741ad8e554380743a4eca, State: Initialized, Role: FOLLOWER
I20260924 19:04:55.294425 19741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e7c26f3205741ad8e554380743a4eca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7c26f3205741ad8e554380743a4eca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37707 } }
I20260924 19:04:55.294624 19741 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e7c26f3205741ad8e554380743a4eca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:04:55.294718 19741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e7c26f3205741ad8e554380743a4eca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:04:55.294835 19741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e7c26f3205741ad8e554380743a4eca [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:04:55.296891 19741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e7c26f3205741ad8e554380743a4eca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7c26f3205741ad8e554380743a4eca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37707 } }
I20260924 19:04:55.297312 19741 leader_election.cc:302] T 00000000000000000000000000000000 P 9e7c26f3205741ad8e554380743a4eca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e7c26f3205741ad8e554380743a4eca; no voters:
I20260924 19:04:55.297942 19741 leader_election.cc:288] T 00000000000000000000000000000000 P 9e7c26f3205741ad8e554380743a4eca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:04:55.298049 19746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e7c26f3205741ad8e554380743a4eca [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:04:55.299320 19746 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e7c26f3205741ad8e554380743a4eca [term 1 LEADER]: Becoming Leader. State: Replica: 9e7c26f3205741ad8e554380743a4eca, State: Running, Role: LEADER
I20260924 19:04:55.299808 19746 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e7c26f3205741ad8e554380743a4eca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7c26f3205741ad8e554380743a4eca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37707 } }
I20260924 19:04:55.300017 19741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e7c26f3205741ad8e554380743a4eca [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:04:55.303326 19748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e7c26f3205741ad8e554380743a4eca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e7c26f3205741ad8e554380743a4eca. Latest consensus state: current_term: 1 leader_uuid: "9e7c26f3205741ad8e554380743a4eca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7c26f3205741ad8e554380743a4eca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37707 } } }
I20260924 19:04:55.303683 19748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e7c26f3205741ad8e554380743a4eca [sys.catalog]: This master's current role is: LEADER
I20260924 19:04:55.304178 19747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e7c26f3205741ad8e554380743a4eca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e7c26f3205741ad8e554380743a4eca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7c26f3205741ad8e554380743a4eca" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37707 } } }
I20260924 19:04:55.304471 19747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e7c26f3205741ad8e554380743a4eca [sys.catalog]: This master's current role is: LEADER
I20260924 19:04:55.304698 19754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:04:55.312417 19754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:04:55.320678 19754 catalog_manager.cc:1347] Generated new cluster ID: 722fd8ec3a12413fab63446b3d7f35f9
I20260924 19:04:55.320806 19754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:04:55.344280 19754 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:04:55.345693 19754 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:04:55.356106 19754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e7c26f3205741ad8e554380743a4eca: Generated new TSK 0
I20260924 19:04:55.357177 19754 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260924 19:04:55.420138 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37707
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:04:55.747774 19767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:55.748046 19767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:55.748143 19767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:55.756177 19767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:55.756286 19767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:55.756397 19767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:04:55.766395 19767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:55.767594 19767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:55.768875 19767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:55.782262 19773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:55.782441 19775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:55.782787 19772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:55.783221 19767 server_base.cc:1034] running on GCE node
I20260924 19:04:55.783936 19767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:55.784947 19767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:55.786171 19767 hybrid_clock.cc:648] HybridClock initialized: now 1790276695786130 us; error 35 us; skew 500 ppm
I20260924 19:04:55.786499 19767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:04:55 dist-test-slave-9fn1 krb5kdc[19661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:04:55.789829 19767 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260924 19:04:55.791752 19767 webserver.cc:460] Webserver started at http://127.0.97.1:40009/ using document root <none> and password file <none>
I20260924 19:04:55.792517 19767 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:55.792618 19767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:55.792894 19767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:04:55.795850 19767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-0/data/instance:
uuid: "d7cd394900634e5a964af7dd966b82a2"
format_stamp: "Formatted at 2026-09-24 19:04:55 on dist-test-slave-9fn1"
server_key: "0797c7e2aeaa994cd03f247628250355"
server_key_iv: "6698a2b65ad0a7751716f6a28538b06d"
server_key_version: "encryptionkey@0"
I20260924 19:04:55.796600 19767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d7cd394900634e5a964af7dd966b82a2"
format_stamp: "Formatted at 2026-09-24 19:04:55 on dist-test-slave-9fn1"
server_key: "0797c7e2aeaa994cd03f247628250355"
server_key_iv: "6698a2b65ad0a7751716f6a28538b06d"
server_key_version: "encryptionkey@0"
I20260924 19:04:55.801736 19767 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 19:04:55.805404 19782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:55.806833 19767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:04:55.806967 19767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "d7cd394900634e5a964af7dd966b82a2"
format_stamp: "Formatted at 2026-09-24 19:04:55 on dist-test-slave-9fn1"
server_key: "0797c7e2aeaa994cd03f247628250355"
server_key_iv: "6698a2b65ad0a7751716f6a28538b06d"
server_key_version: "encryptionkey@0"
I20260924 19:04:55.807147 19767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:55.831113 19767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:55.836036 19767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:55.836295 19767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:55.837212 19767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:04:55.838551 19767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:04:55.838626 19767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:55.838707 19767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:04:55.838752 19767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:55.870563 19767 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33959
I20260924 19:04:55.870646 19894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33959 every 8 connection(s)
I20260924 19:04:55.871811 19767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:04:55.873673 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 19767
I20260924 19:04:55.873870 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-0/wal/instance
I20260924 19:04:55.874330 388 external_mini_cluster.cc:1428] Setting key 2dbdedc88480b366fa150e5c020f297f
Sep 24 19:04:55 dist-test-slave-9fn1 krb5kdc[19661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:04:55.895519 19897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:55.874471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58443 (local address 127.0.97.62:37707)
0924 19:04:55.874930 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:55.874936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:55.875952 (+ 1016us) server_negotiation.cc:407] Connection header received
0924 19:04:55.878795 (+ 2843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:55.878799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:55.878872 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:55.878968 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:55.881009 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:55.881799 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:55.884277 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:55.884792 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:55.884890 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:55.888683 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:55.888691 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:55.888699 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:55.888742 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:55.892277 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:55.892966 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:55.892974 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:55.892980 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:55.893121 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:55.893497 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:55.893534 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:55.893540 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:55.893912 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:55.894086 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:55.894791 (+ 705us) server_negotiation.cc:299] Negotiation successful
0924 19:04:55.895037 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":219,"threads_started":1}
I20260924 19:04:55.896476 19898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:55.875100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37707 (local address 127.0.97.1:58443)
0924 19:04:55.875804 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:55.875849 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:55.878492 (+ 2643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:55.879243 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:55.879257 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:55.880035 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:04:55.880834 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:55.880860 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:55.881948 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:55.881955 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:55.884110 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:55.884128 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:55.885026 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:55.885036 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:55.885362 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:55.886188 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:04:55.886210 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:04:55.888465 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:04:55.892433 (+ 3968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:04:55.892439 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:04:55.892451 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:04:55.892853 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:04:55.893225 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:04:55.893228 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:04:55.893231 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:04:55.893396 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:04:55.894073 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:04:55.894080 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:04:55.894405 (+ 325us) client_negotiation.cc:769] Sending connection context
0924 19:04:55.894657 (+ 252us) client_negotiation.cc:240] Negotiation successful
0924 19:04:55.895061 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":296,"threads_started":1}
I20260924 19:04:55.898209 19895 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37707
I20260924 19:04:55.898532 19895 heartbeater.cc:464] Registering TS with master...
I20260924 19:04:55.899233 19895 heartbeater.cc:511] Master 127.0.97.62:37707 requested a full tablet report, sending...
I20260924 19:04:55.901762 19706 ts_manager.cc:194] Registered new tserver with Master: d7cd394900634e5a964af7dd966b82a2 (127.0.97.1:33959)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260924 19:04:55.903383 19706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:58443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260924 19:04:55.933565 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37707
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:04:56.253885 19901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:56.254160 19901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:56.254308 19901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:56.261488 19901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:56.261658 19901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:56.261837 19901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:04:56.271354 19901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:56.272554 19901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:56.273854 19901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:56.286338 19907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:56.286477 19906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:56.286473 19909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:56.287487 19901 server_base.cc:1034] running on GCE node
I20260924 19:04:56.288065 19901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:56.288930 19901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:56.290130 19901 hybrid_clock.cc:648] HybridClock initialized: now 1790276696290075 us; error 39 us; skew 500 ppm
I20260924 19:04:56.290457 19901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:04:56 dist-test-slave-9fn1 krb5kdc[19661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:04:56.293648 19901 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260924 19:04:56.295159 19901 webserver.cc:460] Webserver started at http://127.0.97.2:45897/ using document root <none> and password file <none>
I20260924 19:04:56.295853 19901 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:56.295997 19901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:56.296298 19901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:04:56.299263 19901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-1/data/instance:
uuid: "974121ed603743f7889e2d48236dccb9"
format_stamp: "Formatted at 2026-09-24 19:04:56 on dist-test-slave-9fn1"
server_key: "3296d9ebfdd8d48ecdf760b22b7eef29"
server_key_iv: "06e2b81ac2cde72bb8986a3625ed97de"
server_key_version: "encryptionkey@0"
I20260924 19:04:56.300041 19901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-1/wal/instance:
uuid: "974121ed603743f7889e2d48236dccb9"
format_stamp: "Formatted at 2026-09-24 19:04:56 on dist-test-slave-9fn1"
server_key: "3296d9ebfdd8d48ecdf760b22b7eef29"
server_key_iv: "06e2b81ac2cde72bb8986a3625ed97de"
server_key_version: "encryptionkey@0"
I20260924 19:04:56.305189 19901 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:04:56.308712 19916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:56.310300 19901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:04:56.310461 19901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "974121ed603743f7889e2d48236dccb9"
format_stamp: "Formatted at 2026-09-24 19:04:56 on dist-test-slave-9fn1"
server_key: "3296d9ebfdd8d48ecdf760b22b7eef29"
server_key_iv: "06e2b81ac2cde72bb8986a3625ed97de"
server_key_version: "encryptionkey@0"
I20260924 19:04:56.310679 19901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:56.325912 19901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:56.329885 19901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:56.330170 19901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:56.331100 19901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:04:56.332404 19901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:04:56.332501 19901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:56.332641 19901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:04:56.332715 19901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:56.364555 19901 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43453
I20260924 19:04:56.364604 20028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43453 every 8 connection(s)
I20260924 19:04:56.365798 19901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:04:56.366285 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 19901
I20260924 19:04:56.366533 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-1/wal/instance
I20260924 19:04:56.367130 388 external_mini_cluster.cc:1428] Setting key 18bcf3c1d7f2fea4e7dd4a980154c503
Sep 24 19:04:56 dist-test-slave-9fn1 krb5kdc[19661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:04:56.386169 19897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:56.368558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38719 (local address 127.0.97.62:37707)
0924 19:04:56.368790 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:56.368797 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:56.370103 (+ 1306us) server_negotiation.cc:407] Connection header received
0924 19:04:56.372068 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:56.372074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:56.372157 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:56.372269 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:56.373770 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.374466 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.375974 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.376402 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.376529 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:56.380357 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:56.380361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:56.380364 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:56.380398 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:56.382899 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:56.383778 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:56.383781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:56.383783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:56.383946 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:56.384429 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:56.384432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:56.384434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:56.384669 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:56.384783 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:56.385746 (+ 963us) server_negotiation.cc:299] Negotiation successful
0924 19:04:56.385918 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 19:04:56.387658 20031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:56.369131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37707 (local address 127.0.97.2:38719)
0924 19:04:56.369961 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:56.370004 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:56.371808 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:56.372509 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:56.372517 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:56.373008 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:04:56.373615 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:56.373634 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.374606 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.374612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:56.375833 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:56.375844 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.376612 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.376618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:56.376833 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:56.377482 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:04:56.377544 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:04:56.380148 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:04:56.383032 (+ 2884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:04:56.383052 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:04:56.383069 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:04:56.383669 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:04:56.384065 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:04:56.384073 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:04:56.384083 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:04:56.384323 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:04:56.384787 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:04:56.384797 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:04:56.385240 (+ 443us) client_negotiation.cc:769] Sending connection context
0924 19:04:56.385608 (+ 368us) client_negotiation.cc:240] Negotiation successful
0924 19:04:56.386192 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":374,"threads_started":1}
I20260924 19:04:56.389127 20029 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37707
I20260924 19:04:56.389472 20029 heartbeater.cc:464] Registering TS with master...
I20260924 19:04:56.390395 20029 heartbeater.cc:511] Master 127.0.97.62:37707 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260924 19:04:56.391959 19706 ts_manager.cc:194] Registered new tserver with Master: 974121ed603743f7889e2d48236dccb9 (127.0.97.2:43453)
I20260924 19:04:56.392930 19706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260924 19:04:56.420704 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37707
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:04:56.739154 20034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:56.739420 20034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:56.739563 20034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:56.746896 20034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:56.747018 20034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:56.747138 20034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:04:56.757014 20034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37707
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:56.758327 20034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:56.759574 20034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:56.773108 20039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:56.773646 20040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:56.773800 20042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:56.774853 20034 server_base.cc:1034] running on GCE node
I20260924 19:04:56.775539 20034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:56.776513 20034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:56.777729 20034 hybrid_clock.cc:648] HybridClock initialized: now 1790276696777682 us; error 39 us; skew 500 ppm
I20260924 19:04:56.778046 20034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:04:56 dist-test-slave-9fn1 krb5kdc[19661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:04:56.781123 20034 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260924 19:04:56.782542 20034 webserver.cc:460] Webserver started at http://127.0.97.3:42715/ using document root <none> and password file <none>
I20260924 19:04:56.783244 20034 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:56.783383 20034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:56.783681 20034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:04:56.786706 20034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-2/data/instance:
uuid: "bebfe2a7f757404b917cf3668fea6d19"
format_stamp: "Formatted at 2026-09-24 19:04:56 on dist-test-slave-9fn1"
server_key: "1b1de6066d4b5a5f76392d67bf9be281"
server_key_iv: "4805c35dba009b662a40bf6aad65aa75"
server_key_version: "encryptionkey@0"
I20260924 19:04:56.787550 20034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-2/wal/instance:
uuid: "bebfe2a7f757404b917cf3668fea6d19"
format_stamp: "Formatted at 2026-09-24 19:04:56 on dist-test-slave-9fn1"
server_key: "1b1de6066d4b5a5f76392d67bf9be281"
server_key_iv: "4805c35dba009b662a40bf6aad65aa75"
server_key_version: "encryptionkey@0"
I20260924 19:04:56.792706 20034 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260924 19:04:56.796130 20049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:56.797367 20034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 19:04:56.797571 20034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "bebfe2a7f757404b917cf3668fea6d19"
format_stamp: "Formatted at 2026-09-24 19:04:56 on dist-test-slave-9fn1"
server_key: "1b1de6066d4b5a5f76392d67bf9be281"
server_key_iv: "4805c35dba009b662a40bf6aad65aa75"
server_key_version: "encryptionkey@0"
I20260924 19:04:56.797763 20034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:56.815379 20034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:56.819264 20034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:56.819550 20034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:56.820479 20034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:04:56.821842 20034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:04:56.821961 20034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:56.822103 20034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:04:56.822173 20034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:56.854291 20034 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35249
I20260924 19:04:56.854336 20161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35249 every 8 connection(s)
I20260924 19:04:56.855507 20034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:04:56.861765 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 20034
I20260924 19:04:56.861948 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-2/wal/instance
I20260924 19:04:56.862365 388 external_mini_cluster.cc:1428] Setting key 3137cc2c476170755c13074d95b1c8ab
Sep 24 19:04:56 dist-test-slave-9fn1 krb5kdc[19661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276696, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:04:56.875113 19897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:56.857941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60289 (local address 127.0.97.62:37707)
0924 19:04:56.858108 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:56.858113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:56.859285 (+ 1172us) server_negotiation.cc:407] Connection header received
0924 19:04:56.861207 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:56.861213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:56.861302 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:56.861411 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:56.863123 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.864108 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.865769 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.866152 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.866284 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:56.869554 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:56.869565 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:56.869569 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:56.869599 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:56.872109 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:56.872819 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:56.872825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:56.872829 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:56.872955 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:56.873368 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:56.873375 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:56.873378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:56.873756 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:56.873927 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:56.874556 (+ 629us) server_negotiation.cc:299] Negotiation successful
0924 19:04:56.874780 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 19:04:56.876111 20164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:56.858414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37707 (local address 127.0.97.3:60289)
0924 19:04:56.859119 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:56.859160 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:56.860938 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:56.861676 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:56.861684 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:56.862270 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:04:56.862938 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:56.862966 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.864250 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.864265 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:56.865624 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:56.865635 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.866376 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.866383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:56.866618 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:56.867207 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:04:56.867229 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:04:56.869279 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:04:56.872245 (+ 2966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:04:56.872253 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:04:56.872275 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:04:56.872696 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:04:56.873060 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:04:56.873066 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:04:56.873069 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:04:56.873253 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:04:56.873897 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:04:56.873904 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:04:56.874199 (+ 295us) client_negotiation.cc:769] Sending connection context
0924 19:04:56.874460 (+ 261us) client_negotiation.cc:240] Negotiation successful
0924 19:04:56.874925 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":283,"threads_started":1}
I20260924 19:04:56.877612 20162 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37707
I20260924 19:04:56.877888 20162 heartbeater.cc:464] Registering TS with master...
I20260924 19:04:56.878542 20162 heartbeater.cc:511] Master 127.0.97.62:37707 requested a full tablet report, sending...
I20260924 19:04:56.879845 19705 ts_manager.cc:194] Registered new tserver with Master: bebfe2a7f757404b917cf3668fea6d19 (127.0.97.3:35249)
I20260924 19:04:56.880716 19705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60289
I20260924 19:04:56.889112 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:04:56.904966 19897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:56.893038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57886 (local address 127.0.97.62:37707)
0924 19:04:56.893188 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:56.893193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:56.893688 (+ 495us) server_negotiation.cc:407] Connection header received
0924 19:04:56.893824 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:56.893827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:56.893887 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:56.893963 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:56.895279 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.895973 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.897259 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.897699 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.897800 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:56.900072 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:56.900077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:56.900080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:56.900108 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:56.902792 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:56.903419 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:56.903422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:56.903424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:56.903527 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:56.903824 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:56.903827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:56.903829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:56.904111 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:56.904293 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:56.904550 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 19:04:56.904698 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 19:04:56.906771 19895 heartbeater.cc:503] Master 127.0.97.62:37707 was elected leader, sending a full tablet report...
I20260924 19:04:56.908780 19705 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:04:56.915990 19705 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:04:56.937757 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:56.929805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59412 (local address 127.0.97.3:35249)
0924 19:04:56.930555 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:56.930561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:56.930584 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:04:56.930660 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:56.930665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:56.930824 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:56.931005 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:56.931628 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.933019 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.935409 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.936513 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.936687 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:56.936908 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:56.937216 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 19:04:56.937372 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":434,"threads_started":1}
I20260924 19:04:56.937939 20178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:56.929166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35249 (local address 127.0.0.1:59412)
0924 19:04:56.929659 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:56.929679 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:56.929790 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:56.931007 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:56.931014 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:56.931042 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:56.931460 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:56.931474 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.933145 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.933153 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:56.935217 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:56.935231 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.936719 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.936725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:56.936949 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:56.937022 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 19:04:56.937170 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 19:04:56.937438 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":70,"thread_start_us":464,"threads_started":1}
I20260924 19:04:56.938771 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:56.927509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33959 (local address 127.0.0.1:59830)
0924 19:04:56.929078 (+ 1569us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:56.929103 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:56.929204 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:56.931056 (+ 1852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:56.931064 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:56.931104 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:56.931697 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:56.931712 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.934163 (+ 2451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.934169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:56.936444 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:56.936458 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.938202 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.938210 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:56.938358 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:56.938381 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 19:04:56.938432 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 19:04:56.938510 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1325,"mutex_wait_us":33,"thread_start_us":356,"threads_started":1}
I20260924 19:04:56.939909 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:56.928054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59830 (local address 127.0.97.1:33959)
0924 19:04:56.930311 (+ 2257us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:56.930323 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:56.930360 (+ 37us) server_negotiation.cc:407] Connection header received
0924 19:04:56.930478 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:56.930489 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:56.930794 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:56.931924 (+ 1130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:56.932239 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.934040 (+ 1801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.936604 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.937961 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.939161 (+ 1200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:56.939385 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:56.939457 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 19:04:56.939597 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2034,"thread_start_us":1871,"threads_started":1}
I20260924 19:04:56.941774 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:56.927366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43453 (local address 127.0.0.1:36190)
0924 19:04:56.928162 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:56.928204 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:56.928352 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:56.934449 (+ 6097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:56.934454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:56.934488 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:56.935032 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:56.935045 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.936938 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.936943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:56.939304 (+ 2361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:56.939316 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.940912 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.940917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:56.941075 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:56.941109 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 19:04:56.941203 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 19:04:56.941351 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":58,"thread_start_us":1335,"threads_started":1}
I20260924 19:04:56.941792 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:56.927510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36190 (local address 127.0.97.2:43453)
0924 19:04:56.933903 (+ 6393us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:56.933912 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:56.933938 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:04:56.934024 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:56.934031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:56.934265 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:56.934440 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:56.935268 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.936771 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.939451 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.940724 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.940869 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:56.941043 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:56.941242 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 19:04:56.941413 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6249,"thread_start_us":6142,"threads_started":1}
I20260924 19:04:56.942212 20097 tablet_service.cc:1467] Processing CreateTablet for tablet 8152a4af3ba1457e89855ca902e0b441 (DEFAULT_TABLE table=test-table [id=f7faa565e04f42b2b71845517e5cc88a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:04:56.943737 20097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8152a4af3ba1457e89855ca902e0b441. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:04:56.943431 19830 tablet_service.cc:1467] Processing CreateTablet for tablet 8152a4af3ba1457e89855ca902e0b441 (DEFAULT_TABLE table=test-table [id=f7faa565e04f42b2b71845517e5cc88a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:04:56.944705 19830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8152a4af3ba1457e89855ca902e0b441. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:04:56.944788 19964 tablet_service.cc:1467] Processing CreateTablet for tablet 8152a4af3ba1457e89855ca902e0b441 (DEFAULT_TABLE table=test-table [id=f7faa565e04f42b2b71845517e5cc88a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:04:56.945935 19964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8152a4af3ba1457e89855ca902e0b441. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:04:56.956152 20180 tablet_bootstrap.cc:492] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9: Bootstrap starting.
I20260924 19:04:56.958104 20181 tablet_bootstrap.cc:492] T 8152a4af3ba1457e89855ca902e0b441 P bebfe2a7f757404b917cf3668fea6d19: Bootstrap starting.
I20260924 19:04:56.960945 20180 tablet_bootstrap.cc:654] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9: Neither blocks nor log segments found. Creating new log.
I20260924 19:04:56.961561 20181 tablet_bootstrap.cc:654] T 8152a4af3ba1457e89855ca902e0b441 P bebfe2a7f757404b917cf3668fea6d19: Neither blocks nor log segments found. Creating new log.
I20260924 19:04:56.962229 20180 log.cc:824] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9: Log is configured to *not* fsync() on all Append() calls
I20260924 19:04:56.962718 20181 log.cc:824] T 8152a4af3ba1457e89855ca902e0b441 P bebfe2a7f757404b917cf3668fea6d19: Log is configured to *not* fsync() on all Append() calls
I20260924 19:04:56.963708 20182 tablet_bootstrap.cc:492] T 8152a4af3ba1457e89855ca902e0b441 P d7cd394900634e5a964af7dd966b82a2: Bootstrap starting.
I20260924 19:04:56.965322 20180 tablet_bootstrap.cc:492] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9: No bootstrap required, opened a new log
I20260924 19:04:56.965617 20180 ts_tablet_manager.cc:1397] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260924 19:04:56.966846 20181 tablet_bootstrap.cc:492] T 8152a4af3ba1457e89855ca902e0b441 P bebfe2a7f757404b917cf3668fea6d19: No bootstrap required, opened a new log
I20260924 19:04:56.967134 20182 tablet_bootstrap.cc:654] T 8152a4af3ba1457e89855ca902e0b441 P d7cd394900634e5a964af7dd966b82a2: Neither blocks nor log segments found. Creating new log.
I20260924 19:04:56.967187 20181 ts_tablet_manager.cc:1397] T 8152a4af3ba1457e89855ca902e0b441 P bebfe2a7f757404b917cf3668fea6d19: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260924 19:04:56.968381 20182 log.cc:824] T 8152a4af3ba1457e89855ca902e0b441 P d7cd394900634e5a964af7dd966b82a2: Log is configured to *not* fsync() on all Append() calls
I20260924 19:04:56.971205 20182 tablet_bootstrap.cc:492] T 8152a4af3ba1457e89855ca902e0b441 P d7cd394900634e5a964af7dd966b82a2: No bootstrap required, opened a new log
I20260924 19:04:56.971450 20182 ts_tablet_manager.cc:1397] T 8152a4af3ba1457e89855ca902e0b441 P d7cd394900634e5a964af7dd966b82a2: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 19:04:56.973865 20180 raft_consensus.cc:348] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974121ed603743f7889e2d48236dccb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43453 } } peers { permanent_uuid: "d7cd394900634e5a964af7dd966b82a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33959 } } peers { permanent_uuid: "bebfe2a7f757404b917cf3668fea6d19" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35249 } }
I20260924 19:04:56.974215 20180 raft_consensus.cc:374] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:04:56.974331 20180 raft_consensus.cc:729] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 974121ed603743f7889e2d48236dccb9, State: Initialized, Role: FOLLOWER
I20260924 19:04:56.975188 20180 consensus_queue.cc:260] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974121ed603743f7889e2d48236dccb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43453 } } peers { permanent_uuid: "d7cd394900634e5a964af7dd966b82a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33959 } } peers { permanent_uuid: "bebfe2a7f757404b917cf3668fea6d19" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35249 } }
I20260924 19:04:56.975694 20186 raft_consensus.cc:482] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:04:56.976016 20186 raft_consensus.cc:504] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974121ed603743f7889e2d48236dccb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43453 } } peers { permanent_uuid: "d7cd394900634e5a964af7dd966b82a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33959 } } peers { permanent_uuid: "bebfe2a7f757404b917cf3668fea6d19" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35249 } }
I20260924 19:04:56.976323 20029 heartbeater.cc:503] Master 127.0.97.62:37707 was elected leader, sending a full tablet report...
I20260924 19:04:56.977438 20181 raft_consensus.cc:348] T 8152a4af3ba1457e89855ca902e0b441 P bebfe2a7f757404b917cf3668fea6d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974121ed603743f7889e2d48236dccb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43453 } } peers { permanent_uuid: "d7cd394900634e5a964af7dd966b82a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33959 } } peers { permanent_uuid: "bebfe2a7f757404b917cf3668fea6d19" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35249 } }
I20260924 19:04:56.977924 20181 raft_consensus.cc:374] T 8152a4af3ba1457e89855ca902e0b441 P bebfe2a7f757404b917cf3668fea6d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:04:56.977941 20186 leader_election.cc:288] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7cd394900634e5a964af7dd966b82a2 (127.0.97.1:33959), bebfe2a7f757404b917cf3668fea6d19 (127.0.97.3:35249)
I20260924 19:04:56.978068 20181 raft_consensus.cc:729] T 8152a4af3ba1457e89855ca902e0b441 P bebfe2a7f757404b917cf3668fea6d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bebfe2a7f757404b917cf3668fea6d19, State: Initialized, Role: FOLLOWER
I20260924 19:04:56.978695 20181 consensus_queue.cc:260] T 8152a4af3ba1457e89855ca902e0b441 P bebfe2a7f757404b917cf3668fea6d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974121ed603743f7889e2d48236dccb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43453 } } peers { permanent_uuid: "d7cd394900634e5a964af7dd966b82a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33959 } } peers { permanent_uuid: "bebfe2a7f757404b917cf3668fea6d19" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35249 } }
I20260924 19:04:56.979521 20180 ts_tablet_manager.cc:1428] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9: Time spent starting tablet: real 0.014s user 0.006s sys 0.004s
I20260924 19:04:56.980515 20162 heartbeater.cc:503] Master 127.0.97.62:37707 was elected leader, sending a full tablet report...
I20260924 19:04:56.981598 20181 ts_tablet_manager.cc:1428] T 8152a4af3ba1457e89855ca902e0b441 P bebfe2a7f757404b917cf3668fea6d19: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260924 19:04:56.988597 20182 raft_consensus.cc:348] T 8152a4af3ba1457e89855ca902e0b441 P d7cd394900634e5a964af7dd966b82a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974121ed603743f7889e2d48236dccb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43453 } } peers { permanent_uuid: "d7cd394900634e5a964af7dd966b82a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33959 } } peers { permanent_uuid: "bebfe2a7f757404b917cf3668fea6d19" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35249 } }
I20260924 19:04:56.989058 20182 raft_consensus.cc:374] T 8152a4af3ba1457e89855ca902e0b441 P d7cd394900634e5a964af7dd966b82a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:04:56.989185 20182 raft_consensus.cc:729] T 8152a4af3ba1457e89855ca902e0b441 P d7cd394900634e5a964af7dd966b82a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7cd394900634e5a964af7dd966b82a2, State: Initialized, Role: FOLLOWER
I20260924 19:04:56.989833 20182 consensus_queue.cc:260] T 8152a4af3ba1457e89855ca902e0b441 P d7cd394900634e5a964af7dd966b82a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974121ed603743f7889e2d48236dccb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43453 } } peers { permanent_uuid: "d7cd394900634e5a964af7dd966b82a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33959 } } peers { permanent_uuid: "bebfe2a7f757404b917cf3668fea6d19" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35249 } }
I20260924 19:04:56.990150 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:56.979510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33959 (local address 127.0.97.2:44723)
0924 19:04:56.982340 (+ 2830us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:56.982364 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:56.982591 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:56.983142 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:56.983146 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:56.983168 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:56.983585 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:56.983602 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.985240 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.985245 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:56.987398 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:56.987414 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.989336 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.989342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:56.989535 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:56.989577 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 19:04:56.989669 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 19:04:56.989815 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2664,"thread_start_us":2519,"threads_started":1}
I20260924 19:04:56.990089 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:56.980831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44723 (local address 127.0.97.1:33959)
0924 19:04:56.981067 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:56.981077 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:56.982464 (+ 1387us) server_negotiation.cc:407] Connection header received
0924 19:04:56.982772 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:56.982778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:56.982866 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:56.982930 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:56.983808 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.985051 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.987573 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.988905 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.989016 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:56.989121 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:56.989712 (+ 591us) server_negotiation.cc:299] Negotiation successful
0924 19:04:56.989861 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 19:04:56.992218 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:56.981801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52453 (local address 127.0.97.3:35249)
0924 19:04:56.982213 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:56.982221 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:56.982861 (+ 640us) server_negotiation.cc:407] Connection header received
0924 19:04:56.983084 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:56.983089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:56.983156 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:56.983216 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:56.984052 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.985115 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.989791 (+ 4676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.991035 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.991155 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:56.991253 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:56.991832 (+ 579us) server_negotiation.cc:299] Negotiation successful
0924 19:04:56.991969 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240}
I20260924 19:04:56.991986 19850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8152a4af3ba1457e89855ca902e0b441" candidate_uuid: "974121ed603743f7889e2d48236dccb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cd394900634e5a964af7dd966b82a2" is_pre_election: true
I20260924 19:04:56.992574 19850 raft_consensus.cc:2454] T 8152a4af3ba1457e89855ca902e0b441 P d7cd394900634e5a964af7dd966b82a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974121ed603743f7889e2d48236dccb9 in term 0.
I20260924 19:04:56.992915 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:56.980513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35249 (local address 127.0.97.2:52453)
0924 19:04:56.982773 (+ 2260us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:56.982788 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:56.982946 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:56.983406 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:56.983412 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:56.983441 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:56.983891 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:56.983903 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.985261 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.985267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:56.989634 (+ 4367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:56.989648 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:56.991415 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:56.991421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:56.991634 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:56.991668 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 19:04:56.992439 (+ 771us) client_negotiation.cc:240] Negotiation successful
0924 19:04:56.992575 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2118,"thread_start_us":1958,"threads_started":1}
I20260924 19:04:56.993425 19917 leader_election.cc:302] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 974121ed603743f7889e2d48236dccb9, d7cd394900634e5a964af7dd966b82a2; no voters:
I20260924 19:04:56.993924 20186 raft_consensus.cc:2781] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:04:56.994058 20186 raft_consensus.cc:482] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:04:56.994159 20186 raft_consensus.cc:3037] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:04:56.994897 20182 ts_tablet_manager.cc:1428] T 8152a4af3ba1457e89855ca902e0b441 P d7cd394900634e5a964af7dd966b82a2: Time spent starting tablet: real 0.023s user 0.017s sys 0.000s
I20260924 19:04:56.995311 20117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8152a4af3ba1457e89855ca902e0b441" candidate_uuid: "974121ed603743f7889e2d48236dccb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebfe2a7f757404b917cf3668fea6d19" is_pre_election: true
I20260924 19:04:56.995735 20117 raft_consensus.cc:2454] T 8152a4af3ba1457e89855ca902e0b441 P bebfe2a7f757404b917cf3668fea6d19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 974121ed603743f7889e2d48236dccb9 in term 0.
I20260924 19:04:56.997272 20186 raft_consensus.cc:504] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974121ed603743f7889e2d48236dccb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43453 } } peers { permanent_uuid: "d7cd394900634e5a964af7dd966b82a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33959 } } peers { permanent_uuid: "bebfe2a7f757404b917cf3668fea6d19" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35249 } }
I20260924 19:04:56.998039 20186 leader_election.cc:288] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9 [CANDIDATE]: Term 1 election: Requested vote from peers d7cd394900634e5a964af7dd966b82a2 (127.0.97.1:33959), bebfe2a7f757404b917cf3668fea6d19 (127.0.97.3:35249)
I20260924 19:04:56.998350 19850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8152a4af3ba1457e89855ca902e0b441" candidate_uuid: "974121ed603743f7889e2d48236dccb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cd394900634e5a964af7dd966b82a2"
I20260924 19:04:56.998533 19850 raft_consensus.cc:3037] T 8152a4af3ba1457e89855ca902e0b441 P d7cd394900634e5a964af7dd966b82a2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:04:56.998618 20117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8152a4af3ba1457e89855ca902e0b441" candidate_uuid: "974121ed603743f7889e2d48236dccb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bebfe2a7f757404b917cf3668fea6d19"
I20260924 19:04:56.998759 20117 raft_consensus.cc:3037] T 8152a4af3ba1457e89855ca902e0b441 P bebfe2a7f757404b917cf3668fea6d19 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:04:57.000676 19850 raft_consensus.cc:2454] T 8152a4af3ba1457e89855ca902e0b441 P d7cd394900634e5a964af7dd966b82a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974121ed603743f7889e2d48236dccb9 in term 1.
I20260924 19:04:57.001034 19917 leader_election.cc:302] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 974121ed603743f7889e2d48236dccb9, d7cd394900634e5a964af7dd966b82a2; no voters:
I20260924 19:04:57.001121 20117 raft_consensus.cc:2454] T 8152a4af3ba1457e89855ca902e0b441 P bebfe2a7f757404b917cf3668fea6d19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 974121ed603743f7889e2d48236dccb9 in term 1.
I20260924 19:04:57.001343 20186 raft_consensus.cc:2781] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:04:57.001533 20186 raft_consensus.cc:686] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9 [term 1 LEADER]: Becoming Leader. State: Replica: 974121ed603743f7889e2d48236dccb9, State: Running, Role: LEADER
I20260924 19:04:57.002264 20186 consensus_queue.cc:237] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974121ed603743f7889e2d48236dccb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43453 } } peers { permanent_uuid: "d7cd394900634e5a964af7dd966b82a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33959 } } peers { permanent_uuid: "bebfe2a7f757404b917cf3668fea6d19" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35249 } }
I20260924 19:04:57.009886 19705 catalog_manager.cc:5495] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 974121ed603743f7889e2d48236dccb9 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "974121ed603743f7889e2d48236dccb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "974121ed603743f7889e2d48236dccb9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43453 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7cd394900634e5a964af7dd966b82a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bebfe2a7f757404b917cf3668fea6d19" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35249 } health_report { overall_health: UNKNOWN } } }
I20260924 19:04:57.042217 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:57.035683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36206 (local address 127.0.97.2:43453)
0924 19:04:57.035920 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:57.035929 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:57.035955 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:04:57.036069 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:57.036077 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:57.036217 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:57.036397 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:04:57.036724 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:57.038087 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:57.039402 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:57.040198 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:57.040344 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:57.040758 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:04:57.041480 (+ 722us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:04:57.041673 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:57.041726 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 19:04:57.041825 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 19:04:57.050163 19850 raft_consensus.cc:1261] T 8152a4af3ba1457e89855ca902e0b441 P d7cd394900634e5a964af7dd966b82a2 [term 1 FOLLOWER]: Refusing update from remote peer 974121ed603743f7889e2d48236dccb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:04:57.050350 20117 raft_consensus.cc:1261] T 8152a4af3ba1457e89855ca902e0b441 P bebfe2a7f757404b917cf3668fea6d19 [term 1 FOLLOWER]: Refusing update from remote peer 974121ed603743f7889e2d48236dccb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:04:57.051261 20187 consensus_queue.cc:1035] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bebfe2a7f757404b917cf3668fea6d19" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:04:57.051721 20186 consensus_queue.cc:1035] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cd394900634e5a964af7dd966b82a2" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:04:57.088331 19705 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:04:57.088665 19705 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:04:57.093837 19705 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e7c26f3205741ad8e554380743a4eca: Sending DeleteTablet for 3 replicas of tablet 8152a4af3ba1457e89855ca902e0b441
I20260924 19:04:57.095014 19964 tablet_service.cc:1514] Processing DeleteTablet for tablet 8152a4af3ba1457e89855ca902e0b441 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:04:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36190
I20260924 19:04:57.095026 19830 tablet_service.cc:1514] Processing DeleteTablet for tablet 8152a4af3ba1457e89855ca902e0b441 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:04:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59830
I20260924 19:04:57.099685 20097 tablet_service.cc:1514] Processing DeleteTablet for tablet 8152a4af3ba1457e89855ca902e0b441 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:04:57 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59412
I20260924 19:04:57.102293 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 19767
I20260924 19:04:57.107658 20202 tablet_replica.cc:330] T 8152a4af3ba1457e89855ca902e0b441 P d7cd394900634e5a964af7dd966b82a2: stopping tablet replica
I20260924 19:04:57.113360 20203 tablet_replica.cc:330] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9: stopping tablet replica
I20260924 19:04:57.113559 20202 raft_consensus.cc:2229] T 8152a4af3ba1457e89855ca902e0b441 P d7cd394900634e5a964af7dd966b82a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:04:57.113854 20203 raft_consensus.cc:2229] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:04:57.114570 20203 raft_consensus.cc:2258] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:04:57.114578 20202 raft_consensus.cc:2258] T 8152a4af3ba1457e89855ca902e0b441 P d7cd394900634e5a964af7dd966b82a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:04:57.115687 20204 tablet_replica.cc:330] T 8152a4af3ba1457e89855ca902e0b441 P bebfe2a7f757404b917cf3668fea6d19: stopping tablet replica
I20260924 19:04:57.116113 20204 raft_consensus.cc:2229] T 8152a4af3ba1457e89855ca902e0b441 P bebfe2a7f757404b917cf3668fea6d19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:04:57.117010 20203 ts_tablet_manager.cc:1905] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:04:57.117146 20204 raft_consensus.cc:2258] T 8152a4af3ba1457e89855ca902e0b441 P bebfe2a7f757404b917cf3668fea6d19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:04:57.117687 20202 ts_tablet_manager.cc:1905] T 8152a4af3ba1457e89855ca902e0b441 P d7cd394900634e5a964af7dd966b82a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:04:57.120757 20204 ts_tablet_manager.cc:1905] T 8152a4af3ba1457e89855ca902e0b441 P bebfe2a7f757404b917cf3668fea6d19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:04:57.124765 20202 ts_tablet_manager.cc:1918] T 8152a4af3ba1457e89855ca902e0b441 P d7cd394900634e5a964af7dd966b82a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:04:57.124972 20202 log.cc:1196] T 8152a4af3ba1457e89855ca902e0b441 P d7cd394900634e5a964af7dd966b82a2: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-0/wal/wals/8152a4af3ba1457e89855ca902e0b441
I20260924 19:04:57.125615 20202 ts_tablet_manager.cc:1939] T 8152a4af3ba1457e89855ca902e0b441 P d7cd394900634e5a964af7dd966b82a2: Deleting consensus metadata
I20260924 19:04:57.127535 19690 catalog_manager.cc:4841] TS d7cd394900634e5a964af7dd966b82a2 (127.0.97.1:33959): tablet 8152a4af3ba1457e89855ca902e0b441 (table test-table [id=f7faa565e04f42b2b71845517e5cc88a]) successfully deleted
Sep 24 19:04:57 dist-test-slave-9fn1 krb5kdc[19661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260924 19:04:57.136293 20204 ts_tablet_manager.cc:1918] T 8152a4af3ba1457e89855ca902e0b441 P bebfe2a7f757404b917cf3668fea6d19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:04:57.136571 20204 log.cc:1196] T 8152a4af3ba1457e89855ca902e0b441 P bebfe2a7f757404b917cf3668fea6d19: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-2/wal/wals/8152a4af3ba1457e89855ca902e0b441
I20260924 19:04:57.137184 20204 ts_tablet_manager.cc:1939] T 8152a4af3ba1457e89855ca902e0b441 P bebfe2a7f757404b917cf3668fea6d19: Deleting consensus metadata
I20260924 19:04:57.138937 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:57.103936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59842 (local address 127.0.97.1:33959)
0924 19:04:57.104134 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:57.104142 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:57.112775 (+ 8633us) server_negotiation.cc:407] Connection header received
0924 19:04:57.112842 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:57.112850 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:57.112941 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:57.113050 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:57.121937 (+ 8887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:57.123117 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:57.125106 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:57.125874 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:57.125972 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:57.131120 (+ 5148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:57.131134 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:57.131151 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:57.131187 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:57.134864 (+ 3677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:57.135735 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:57.135756 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:57.135759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:57.135901 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:57.136373 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:57.136381 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:57.136386 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:57.136887 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:57.137190 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:57.138379 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0924 19:04:57.138580 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 19:04:57.139129 19691 catalog_manager.cc:4841] TS bebfe2a7f757404b917cf3668fea6d19 (127.0.97.3:35249): tablet 8152a4af3ba1457e89855ca902e0b441 (table test-table [id=f7faa565e04f42b2b71845517e5cc88a]) successfully deleted
I20260924 19:04:57.139870 19890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:04:57.143726 20203 ts_tablet_manager.cc:1918] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:04:57.143944 20203 log.cc:1196] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276386554690-388-0/minicluster-data/ts-1/wal/wals/8152a4af3ba1457e89855ca902e0b441
I20260924 19:04:57.144534 20203 ts_tablet_manager.cc:1939] T 8152a4af3ba1457e89855ca902e0b441 P 974121ed603743f7889e2d48236dccb9: Deleting consensus metadata
I20260924 19:04:57.146355 19691 catalog_manager.cc:4841] TS 974121ed603743f7889e2d48236dccb9 (127.0.97.2:43453): tablet 8152a4af3ba1457e89855ca902e0b441 (table test-table [id=f7faa565e04f42b2b71845517e5cc88a]) successfully deleted
I20260924 19:04:57.297569 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 19767
I20260924 19:04:57.315943 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 19901
Sep 24 19:04:57 dist-test-slave-9fn1 krb5kdc[19661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260924 19:04:57.331568 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:57.316774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36208 (local address 127.0.97.2:43453)
0924 19:04:57.316941 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:57.316946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:57.317020 (+ 74us) server_negotiation.cc:407] Connection header received
0924 19:04:57.317218 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:57.317223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:57.317296 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:57.317409 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:57.318645 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:57.319440 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:57.320701 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:57.321138 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:57.321263 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:57.325545 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:57.325567 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:57.325583 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:57.325618 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:57.328488 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:57.329466 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:57.329475 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:57.329479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:57.329677 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:57.330035 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:57.330041 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:57.330044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:57.330402 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:57.330745 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:57.330998 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 19:04:57.331213 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:04:57.332252 20024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:04:57.443665 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 19901
I20260924 19:04:57.461674 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 20034
Sep 24 19:04:57 dist-test-slave-9fn1 krb5kdc[19661](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260924 19:04:57.476410 20179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:57.462461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59424 (local address 127.0.97.3:35249)
0924 19:04:57.462643 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:57.462648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:57.462666 (+ 18us) server_negotiation.cc:407] Connection header received
0924 19:04:57.462821 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:57.462825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:57.462889 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:57.462993 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:57.464264 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:57.464997 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:57.466296 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:57.466671 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:57.466763 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:57.471094 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:57.471108 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:57.471124 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:04:57.471163 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:57.473741 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:57.474420 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:57.474428 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:57.474430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:57.474570 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:04:57.474931 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:04:57.474938 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:04:57.474941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:04:57.475267 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:57.475566 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:57.475857 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 19:04:57.476083 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 19:04:57.477061 20157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:04:57.567698 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 20034
I20260924 19:04:57.584537 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 19674
I20260924 19:04:57.585644 19736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:04:57.694979 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 19674
2026-09-24T19:04:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4275 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T19:04:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:04:57Z Disabled control of system clock
I20260924 19:04:57.769385 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33605
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:04:58.090909 20214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:58.091187 20214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:58.091326 20214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:58.098472 20214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:04:58.098583 20214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:58.098673 20214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:58.098769 20214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:04:58.098843 20214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:04:58.107935 20214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33605
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:58.109313 20214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:58.111097 20214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:58.122568 20222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:58.122640 20219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:58.122668 20220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:58.123504 20214 server_base.cc:1034] running on GCE node
I20260924 19:04:58.124212 20214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:58.125602 20214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:58.126803 20214 hybrid_clock.cc:648] HybridClock initialized: now 1790276698126758 us; error 35 us; skew 500 ppm
I20260924 19:04:58.127099 20214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:04:58.129129 20214 webserver.cc:460] Webserver started at http://127.0.97.62:46565/ using document root <none> and password file <none>
I20260924 19:04:58.129828 20214 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:58.129913 20214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:58.130159 20214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:04:58.132820 20214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "8504e3bcaaaa41d1aa1120be61261835"
format_stamp: "Formatted at 2026-09-24 19:04:58 on dist-test-slave-9fn1"
server_key: "794e896c6dc06863152b8158dec2ae42"
server_key_iv: "e229f342023114dc8ab8d108e6906def"
server_key_version: "encryptionkey@0"
I20260924 19:04:58.133596 20214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "8504e3bcaaaa41d1aa1120be61261835"
format_stamp: "Formatted at 2026-09-24 19:04:58 on dist-test-slave-9fn1"
server_key: "794e896c6dc06863152b8158dec2ae42"
server_key_iv: "e229f342023114dc8ab8d108e6906def"
server_key_version: "encryptionkey@0"
I20260924 19:04:58.138350 20214 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 19:04:58.141465 20228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:58.142705 20214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:04:58.142848 20214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "8504e3bcaaaa41d1aa1120be61261835"
format_stamp: "Formatted at 2026-09-24 19:04:58 on dist-test-slave-9fn1"
server_key: "794e896c6dc06863152b8158dec2ae42"
server_key_iv: "e229f342023114dc8ab8d108e6906def"
server_key_version: "encryptionkey@0"
I20260924 19:04:58.143007 20214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:58.170734 20214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:58.171588 20214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:58.171839 20214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:58.189618 20214 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33605
I20260924 19:04:58.189651 20279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33605 every 8 connection(s)
I20260924 19:04:58.190889 20214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:04:58.194577 20280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:04:58.201018 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 20214
I20260924 19:04:58.201179 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:04:58.201699 388 external_mini_cluster.cc:1428] Setting key 5364a34647ea42493f01ab72f4e88468
I20260924 19:04:58.205317 20280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8504e3bcaaaa41d1aa1120be61261835: Bootstrap starting.
I20260924 19:04:58.209280 20280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8504e3bcaaaa41d1aa1120be61261835: Neither blocks nor log segments found. Creating new log.
I20260924 19:04:58.210552 20280 log.cc:824] T 00000000000000000000000000000000 P 8504e3bcaaaa41d1aa1120be61261835: Log is configured to *not* fsync() on all Append() calls
I20260924 19:04:58.213665 20280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8504e3bcaaaa41d1aa1120be61261835: No bootstrap required, opened a new log
I20260924 19:04:58.217391 20283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:58.203587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56810 (local address 127.0.97.62:33605)
0924 19:04:58.204579 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:58.204601 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:58.204650 (+ 49us) server_negotiation.cc:407] Connection header received
0924 19:04:58.207029 (+ 2379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:58.207065 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:58.207693 (+ 628us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:58.208105 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:58.208665 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:58.210721 (+ 2056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:58.212376 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:58.213408 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:58.213614 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:58.214723 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:58.214734 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:58.214752 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:04:58.215444 (+ 692us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:58.215533 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:04:58.215562 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:58.215681 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:58.215836 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 19:04:58.216147 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":303,"threads_started":1}
I20260924 19:04:58.222787 20280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8504e3bcaaaa41d1aa1120be61261835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8504e3bcaaaa41d1aa1120be61261835" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33605 } }
I20260924 19:04:58.223084 20280 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8504e3bcaaaa41d1aa1120be61261835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:04:58.223151 20280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8504e3bcaaaa41d1aa1120be61261835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8504e3bcaaaa41d1aa1120be61261835, State: Initialized, Role: FOLLOWER
I20260924 19:04:58.223616 20280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8504e3bcaaaa41d1aa1120be61261835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8504e3bcaaaa41d1aa1120be61261835" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33605 } }
I20260924 19:04:58.223812 20280 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8504e3bcaaaa41d1aa1120be61261835 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:04:58.223904 20280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8504e3bcaaaa41d1aa1120be61261835 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:04:58.224035 20280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8504e3bcaaaa41d1aa1120be61261835 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:04:58.226279 20280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8504e3bcaaaa41d1aa1120be61261835 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8504e3bcaaaa41d1aa1120be61261835" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33605 } }
I20260924 19:04:58.226706 20280 leader_election.cc:302] T 00000000000000000000000000000000 P 8504e3bcaaaa41d1aa1120be61261835 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8504e3bcaaaa41d1aa1120be61261835; no voters:
I20260924 19:04:58.227151 20280 leader_election.cc:288] T 00000000000000000000000000000000 P 8504e3bcaaaa41d1aa1120be61261835 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:04:58.227293 20285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8504e3bcaaaa41d1aa1120be61261835 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:04:58.227998 20285 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8504e3bcaaaa41d1aa1120be61261835 [term 1 LEADER]: Becoming Leader. State: Replica: 8504e3bcaaaa41d1aa1120be61261835, State: Running, Role: LEADER
I20260924 19:04:58.228515 20285 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8504e3bcaaaa41d1aa1120be61261835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8504e3bcaaaa41d1aa1120be61261835" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33605 } }
I20260924 19:04:58.229125 20280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8504e3bcaaaa41d1aa1120be61261835 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:04:58.231155 20287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8504e3bcaaaa41d1aa1120be61261835 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8504e3bcaaaa41d1aa1120be61261835. Latest consensus state: current_term: 1 leader_uuid: "8504e3bcaaaa41d1aa1120be61261835" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8504e3bcaaaa41d1aa1120be61261835" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33605 } } }
I20260924 19:04:58.231518 20287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8504e3bcaaaa41d1aa1120be61261835 [sys.catalog]: This master's current role is: LEADER
I20260924 19:04:58.232174 20286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8504e3bcaaaa41d1aa1120be61261835 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8504e3bcaaaa41d1aa1120be61261835" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8504e3bcaaaa41d1aa1120be61261835" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33605 } } }
I20260924 19:04:58.232654 20286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8504e3bcaaaa41d1aa1120be61261835 [sys.catalog]: This master's current role is: LEADER
I20260924 19:04:58.233707 20290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:04:58.239179 20290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:04:58.247521 20290 catalog_manager.cc:1347] Generated new cluster ID: a511e71041df44efacf06af66af749b2
I20260924 19:04:58.247658 20290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:04:58.271507 20290 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:04:58.273046 20290 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:04:58.282207 20290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8504e3bcaaaa41d1aa1120be61261835: Generated new TSK 0
I20260924 19:04:58.283282 20290 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:04:58.298857 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33605
--builtin_ntp_servers=127.0.97.20:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:04:58.623647 20304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:58.623956 20304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:58.624101 20304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:58.631314 20304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:58.631444 20304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:58.631606 20304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260924 19:04:58.640491 20304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:58.641880 20304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:58.643602 20304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:58.656836 20309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:58.656868 20310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:58.656961 20312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:58.657816 20304 server_base.cc:1034] running on GCE node
I20260924 19:04:58.658347 20304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:58.659194 20304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:58.660395 20304 hybrid_clock.cc:648] HybridClock initialized: now 1790276698660343 us; error 41 us; skew 500 ppm
I20260924 19:04:58.660708 20304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:04:58.662844 20304 webserver.cc:460] Webserver started at http://127.0.97.1:40141/ using document root <none> and password file <none>
I20260924 19:04:58.663542 20304 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:58.663661 20304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:58.663997 20304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:04:58.666778 20304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-0/data/instance:
uuid: "d1096652909d44c1aa25ee0d4e060c44"
format_stamp: "Formatted at 2026-09-24 19:04:58 on dist-test-slave-9fn1"
server_key: "a660a084e23821db64cf778a8105638b"
server_key_iv: "f030eb35137c7e372f3e2e65ec554814"
server_key_version: "encryptionkey@0"
I20260924 19:04:58.667502 20304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d1096652909d44c1aa25ee0d4e060c44"
format_stamp: "Formatted at 2026-09-24 19:04:58 on dist-test-slave-9fn1"
server_key: "a660a084e23821db64cf778a8105638b"
server_key_iv: "f030eb35137c7e372f3e2e65ec554814"
server_key_version: "encryptionkey@0"
I20260924 19:04:58.672319 20304 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 19:04:58.675707 20318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:58.676980 20304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:04:58.677155 20304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-0/wal
uuid: "d1096652909d44c1aa25ee0d4e060c44"
format_stamp: "Formatted at 2026-09-24 19:04:58 on dist-test-slave-9fn1"
server_key: "a660a084e23821db64cf778a8105638b"
server_key_iv: "f030eb35137c7e372f3e2e65ec554814"
server_key_version: "encryptionkey@0"
I20260924 19:04:58.677369 20304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:58.709160 20304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:58.710034 20304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:58.710316 20304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:58.711248 20304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:04:58.712853 20304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:04:58.712978 20304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:58.713097 20304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:04:58.713181 20304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:58.745556 20304 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44233
I20260924 19:04:58.745643 20430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44233 every 8 connection(s)
I20260924 19:04:58.746953 20304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-0/data/info.pb
I20260924 19:04:58.755276 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 20304
I20260924 19:04:58.755463 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-0/wal/instance
I20260924 19:04:58.755872 388 external_mini_cluster.cc:1428] Setting key 8c4a8aaec8120bf14ee55da0ab2f49a1
I20260924 19:04:58.758569 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33605
--builtin_ntp_servers=127.0.97.20:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 19:04:58.766281 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:58.749360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55481 (local address 127.0.97.62:33605)
0924 19:04:58.749812 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:58.749816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:58.750881 (+ 1065us) server_negotiation.cc:407] Connection header received
0924 19:04:58.752673 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:58.752678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:58.752766 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:58.752880 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:58.753955 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:58.754685 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:58.756344 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:58.756883 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:58.757011 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:58.765436 (+ 8425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:58.765442 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:58.765446 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:04:58.765472 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:58.765563 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:04:58.765585 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:58.765662 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:58.765949 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 19:04:58.766033 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":256,"threads_started":1}
I20260924 19:04:58.767853 20434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:58.749878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33605 (local address 127.0.97.1:55481)
0924 19:04:58.750719 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:58.750771 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:58.752408 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:58.753104 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:58.753112 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:58.753195 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:04:58.753790 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:58.753807 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:58.754837 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:58.754841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:58.756207 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:58.756222 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:58.757108 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:58.757116 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:58.757347 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:58.765028 (+ 7681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:04:58.765034 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:04:58.765096 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:04:58.765209 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:04:58.765695 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:04:58.765708 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:04:58.765720 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 19:04:58.765919 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 19:04:58.766369 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":246,"threads_started":1}
I20260924 19:04:58.769371 20431 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33605
I20260924 19:04:58.769701 20431 heartbeater.cc:464] Registering TS with master...
I20260924 19:04:58.770433 20431 heartbeater.cc:511] Master 127.0.97.62:33605 requested a full tablet report, sending...
I20260924 19:04:58.772457 20245 ts_manager.cc:194] Registered new tserver with Master: d1096652909d44c1aa25ee0d4e060c44 (127.0.97.1:44233)
I20260924 19:04:58.773993 20245 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:55481
W20260924 19:04:59.093484 20435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:59.093814 20435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:59.093972 20435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:59.101032 20435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:59.101169 20435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:59.101351 20435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260924 19:04:59.109999 20435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:59.111238 20435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:59.112927 20435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:59.125689 20440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:59.125679 20441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:59.125660 20443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:59.126679 20435 server_base.cc:1034] running on GCE node
I20260924 19:04:59.127391 20435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:59.128274 20435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:59.129540 20435 hybrid_clock.cc:648] HybridClock initialized: now 1790276699129451 us; error 52 us; skew 500 ppm
I20260924 19:04:59.129858 20435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:04:59.132051 20435 webserver.cc:460] Webserver started at http://127.0.97.2:40369/ using document root <none> and password file <none>
I20260924 19:04:59.132679 20435 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:59.132759 20435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:59.133009 20435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:04:59.135850 20435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-1/data/instance:
uuid: "1443227823bb45f9afd194a5d0d303b4"
format_stamp: "Formatted at 2026-09-24 19:04:59 on dist-test-slave-9fn1"
server_key: "4fd61a6e1dd0a0936a635c23da56bb85"
server_key_iv: "7da84d8a40e6ac620a1b263b7d6ef157"
server_key_version: "encryptionkey@0"
I20260924 19:04:59.136551 20435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1443227823bb45f9afd194a5d0d303b4"
format_stamp: "Formatted at 2026-09-24 19:04:59 on dist-test-slave-9fn1"
server_key: "4fd61a6e1dd0a0936a635c23da56bb85"
server_key_iv: "7da84d8a40e6ac620a1b263b7d6ef157"
server_key_version: "encryptionkey@0"
I20260924 19:04:59.141323 20435 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260924 19:04:59.144451 20449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:59.145709 20435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260924 19:04:59.145866 20435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-1/wal
uuid: "1443227823bb45f9afd194a5d0d303b4"
format_stamp: "Formatted at 2026-09-24 19:04:59 on dist-test-slave-9fn1"
server_key: "4fd61a6e1dd0a0936a635c23da56bb85"
server_key_iv: "7da84d8a40e6ac620a1b263b7d6ef157"
server_key_version: "encryptionkey@0"
I20260924 19:04:59.146070 20435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:59.159286 20435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:59.160030 20435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:59.160300 20435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:59.161194 20435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:04:59.162607 20435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:04:59.162725 20435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:59.162844 20435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:04:59.162953 20435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:59.194953 20435 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40927
I20260924 19:04:59.194989 20561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40927 every 8 connection(s)
I20260924 19:04:59.196269 20435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-1/data/info.pb
I20260924 19:04:59.202267 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 20435
I20260924 19:04:59.202431 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-1/wal/instance
I20260924 19:04:59.202991 388 external_mini_cluster.cc:1428] Setting key 65fc304437fa8ab940497609f07c91af
I20260924 19:04:59.205946 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33605
--builtin_ntp_servers=127.0.97.20:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 19:04:59.208835 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:59.198884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59603 (local address 127.0.97.62:33605)
0924 19:04:59.199050 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:59.199054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:59.200530 (+ 1476us) server_negotiation.cc:407] Connection header received
0924 19:04:59.202335 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:59.202340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:59.202427 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:59.202526 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:59.203540 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.204393 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.206166 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.206595 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.206668 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:59.208069 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:59.208075 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:59.208080 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:04:59.208100 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:59.208135 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:04:59.208146 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:59.208240 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:59.208449 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 19:04:59.208543 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 19:04:59.209870 20564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:59.199481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33605 (local address 127.0.97.2:59603)
0924 19:04:59.200380 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:59.200418 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:59.202086 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:59.202763 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:59.202771 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:59.202854 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:04:59.203372 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:59.203388 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.204552 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.204563 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:59.205975 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:59.205986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.206915 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.206919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:59.207101 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:59.207777 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:04:59.207779 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:04:59.207822 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:04:59.207891 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:04:59.208250 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:04:59.208262 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:04:59.208275 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 19:04:59.208439 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 19:04:59.208815 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":304,"threads_started":1}
I20260924 19:04:59.211205 20562 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33605
I20260924 19:04:59.211493 20562 heartbeater.cc:464] Registering TS with master...
I20260924 19:04:59.212179 20562 heartbeater.cc:511] Master 127.0.97.62:33605 requested a full tablet report, sending...
I20260924 19:04:59.213532 20245 ts_manager.cc:194] Registered new tserver with Master: 1443227823bb45f9afd194a5d0d303b4 (127.0.97.2:40927)
I20260924 19:04:59.214293 20245 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:59603
W20260924 19:04:59.522428 20565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:04:59.522713 20565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:04:59.522819 20565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:04:59.529956 20565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:04:59.530063 20565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:04:59.530237 20565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260924 19:04:59.539531 20565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33605
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:04:59.540809 20565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:04:59.542671 20565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:04:59.555124 20570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:59.555171 20571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:04:59.555192 20573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:04:59.556001 20565 server_base.cc:1034] running on GCE node
I20260924 19:04:59.556655 20565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:04:59.557631 20565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:04:59.558849 20565 hybrid_clock.cc:648] HybridClock initialized: now 1790276699558803 us; error 55 us; skew 500 ppm
I20260924 19:04:59.559152 20565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:04:59.561259 20565 webserver.cc:460] Webserver started at http://127.0.97.3:43691/ using document root <none> and password file <none>
I20260924 19:04:59.561960 20565 fs_manager.cc:359] Metadata directory not provided
I20260924 19:04:59.562080 20565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:04:59.562386 20565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:04:59.565126 20565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-2/data/instance:
uuid: "469fb42bacce4f3b9265b7e9f4c67e69"
format_stamp: "Formatted at 2026-09-24 19:04:59 on dist-test-slave-9fn1"
server_key: "17ac75c63aad9d91775d22b68d4d1925"
server_key_iv: "ab731d7813b8bccd5c59db7d75d6e1f4"
server_key_version: "encryptionkey@0"
I20260924 19:04:59.565927 20565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-2/wal/instance:
uuid: "469fb42bacce4f3b9265b7e9f4c67e69"
format_stamp: "Formatted at 2026-09-24 19:04:59 on dist-test-slave-9fn1"
server_key: "17ac75c63aad9d91775d22b68d4d1925"
server_key_iv: "ab731d7813b8bccd5c59db7d75d6e1f4"
server_key_version: "encryptionkey@0"
I20260924 19:04:59.570616 20565 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 19:04:59.573868 20579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:59.575060 20565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:04:59.575214 20565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-2/wal
uuid: "469fb42bacce4f3b9265b7e9f4c67e69"
format_stamp: "Formatted at 2026-09-24 19:04:59 on dist-test-slave-9fn1"
server_key: "17ac75c63aad9d91775d22b68d4d1925"
server_key_iv: "ab731d7813b8bccd5c59db7d75d6e1f4"
server_key_version: "encryptionkey@0"
I20260924 19:04:59.575416 20565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:04:59.599704 20565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:04:59.600569 20565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:04:59.600852 20565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:04:59.601810 20565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:04:59.603214 20565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:04:59.603330 20565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:59.603456 20565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:04:59.603544 20565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:04:59.635484 20565 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46583
I20260924 19:04:59.635564 20691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46583 every 8 connection(s)
I20260924 19:04:59.636757 20565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-2/data/info.pb
I20260924 19:04:59.640681 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 20565
I20260924 19:04:59.640841 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-2/wal/instance
I20260924 19:04:59.641278 388 external_mini_cluster.cc:1428] Setting key 3d865fec1087b7bb5d77089ca767330f
I20260924 19:04:59.649331 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:59.639227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33097 (local address 127.0.97.62:33605)
0924 19:04:59.639395 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:59.639399 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:59.640726 (+ 1327us) server_negotiation.cc:407] Connection header received
0924 19:04:59.643009 (+ 2283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:59.643014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:59.643103 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:59.643204 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:59.644343 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.645065 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.646669 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.647089 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.647214 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:59.648548 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:59.648552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:59.648555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:04:59.648572 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:59.648601 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:04:59.648624 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:59.648711 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:59.648912 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 19:04:59.649001 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 19:04:59.650259 20694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:59.639737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33605 (local address 127.0.97.3:33097)
0924 19:04:59.640583 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:59.640617 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:59.642725 (+ 2108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:59.643430 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:59.643438 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:59.643532 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:04:59.644188 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:59.644205 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.645202 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.645208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:59.646531 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:59.646547 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.647299 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.647307 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:59.647525 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:59.648237 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:04:59.648240 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:04:59.648275 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:04:59.648349 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:04:59.648711 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:04:59.648723 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:04:59.648733 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 19:04:59.648903 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 19:04:59.649278 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":250,"threads_started":1}
I20260924 19:04:59.651475 20692 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33605
I20260924 19:04:59.651696 20692 heartbeater.cc:464] Registering TS with master...
I20260924 19:04:59.652290 20692 heartbeater.cc:511] Master 127.0.97.62:33605 requested a full tablet report, sending...
I20260924 19:04:59.653699 20245 ts_manager.cc:194] Registered new tserver with Master: 469fb42bacce4f3b9265b7e9f4c67e69 (127.0.97.3:46583)
I20260924 19:04:59.654685 20245 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:33097
I20260924 19:04:59.656059 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:04:59.664877 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:59.659851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56820 (local address 127.0.97.62:33605)
0924 19:04:59.660090 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:59.660096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:59.660349 (+ 253us) server_negotiation.cc:407] Connection header received
0924 19:04:59.660480 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:59.660484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:59.660548 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:59.660626 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:59.661037 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.661739 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.662911 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.663306 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.663391 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:59.664276 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:59.664281 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:59.664284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:04:59.664299 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:59.664326 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:04:59.664336 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:59.664400 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:59.664505 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 19:04:59.664568 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 19:04:59.668165 20245 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:04:59.674942 20245 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:04:59.711666 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:59.699734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60090 (local address 127.0.97.2:40927)
0924 19:04:59.702637 (+ 2903us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:59.702649 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:59.703352 (+ 703us) server_negotiation.cc:407] Connection header received
0924 19:04:59.703581 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:59.703593 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:59.703860 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:59.704046 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:59.704662 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.706066 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.709091 (+ 3025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.710496 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.710654 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:59.710961 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:59.711152 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 19:04:59.711314 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2695,"thread_start_us":2524,"threads_started":1}
I20260924 19:04:59.711733 20704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:59.701733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40927 (local address 127.0.0.1:60090)
0924 19:04:59.703212 (+ 1479us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:59.703228 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:59.703362 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:59.704068 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:59.704075 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:59.704104 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:59.704444 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:59.704457 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.706214 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.706218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:59.708905 (+ 2687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:59.708919 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.710695 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.710704 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:59.710908 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:59.710956 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 19:04:59.711099 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 19:04:59.711301 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1364,"mutex_wait_us":205,"thread_start_us":923,"threads_started":1}
I20260924 19:04:59.711889 20705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:59.686167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46583 (local address 127.0.0.1:36678)
0924 19:04:59.697851 (+ 11684us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:59.697891 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:59.698114 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:59.704476 (+ 6362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:59.704480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:59.704512 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:59.705093 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:59.705108 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.706986 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.706994 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:59.709191 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:59.709200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.711148 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.711153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:59.711313 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:59.711341 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 19:04:59.711424 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 19:04:59.711536 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11367,"mutex_wait_us":24,"thread_start_us":16794,"threads_started":1}
I20260924 19:04:59.712467 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:59.686528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36678 (local address 127.0.97.3:46583)
0924 19:04:59.703907 (+ 17379us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:59.703917 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:59.703946 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:04:59.704035 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:59.704042 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:59.704267 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:59.704472 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:59.705284 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.706821 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.709345 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.710622 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.710752 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:59.711028 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:59.712089 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0924 19:04:59.712241 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17211,"thread_start_us":17061,"threads_started":1}
I20260924 19:04:59.712882 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:59.686227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44233 (local address 127.0.0.1:37190)
0924 19:04:59.702824 (+ 16597us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:59.702839 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:59.702946 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:59.704532 (+ 1586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:59.704538 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:59.704565 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:59.705122 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:59.705140 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.706827 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.706833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:59.708653 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:59.708665 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.711520 (+ 2855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.711526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:59.711741 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:59.711777 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 19:04:59.711885 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 19:04:59.711994 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16478,"mutex_wait_us":169,"thread_start_us":12138,"threads_started":1}
I20260924 19:04:59.713677 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:59.686594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37190 (local address 127.0.97.1:44233)
0924 19:04:59.700967 (+ 14373us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:59.700974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:59.702931 (+ 1957us) server_negotiation.cc:407] Connection header received
0924 19:04:59.703909 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:59.703917 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:59.704146 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:59.704322 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:59.705309 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.706641 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.708853 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.711016 (+ 2163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.711164 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:59.711387 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:59.713232 (+ 1845us) server_negotiation.cc:299] Negotiation successful
0924 19:04:59.713353 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14198,"thread_start_us":14088,"threads_started":1}
I20260924 19:04:59.716018 20627 tablet_service.cc:1467] Processing CreateTablet for tablet d4f405eb3acf4ee5ab0c48c65fb8c69a (DEFAULT_TABLE table=test-table [id=a753ecc0ca1a484a8f80380847d7fad5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:04:59.716018 20497 tablet_service.cc:1467] Processing CreateTablet for tablet d4f405eb3acf4ee5ab0c48c65fb8c69a (DEFAULT_TABLE table=test-table [id=a753ecc0ca1a484a8f80380847d7fad5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:04:59.716554 20366 tablet_service.cc:1467] Processing CreateTablet for tablet d4f405eb3acf4ee5ab0c48c65fb8c69a (DEFAULT_TABLE table=test-table [id=a753ecc0ca1a484a8f80380847d7fad5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:04:59.717444 20627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4f405eb3acf4ee5ab0c48c65fb8c69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:04:59.717444 20497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4f405eb3acf4ee5ab0c48c65fb8c69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:04:59.717693 20366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4f405eb3acf4ee5ab0c48c65fb8c69a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:04:59.729020 20710 tablet_bootstrap.cc:492] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44: Bootstrap starting.
I20260924 19:04:59.731915 20711 tablet_bootstrap.cc:492] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 1443227823bb45f9afd194a5d0d303b4: Bootstrap starting.
I20260924 19:04:59.733819 20710 tablet_bootstrap.cc:654] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44: Neither blocks nor log segments found. Creating new log.
I20260924 19:04:59.735162 20710 log.cc:824] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44: Log is configured to *not* fsync() on all Append() calls
I20260924 19:04:59.735662 20711 tablet_bootstrap.cc:654] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 1443227823bb45f9afd194a5d0d303b4: Neither blocks nor log segments found. Creating new log.
I20260924 19:04:59.736619 20712 tablet_bootstrap.cc:492] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 469fb42bacce4f3b9265b7e9f4c67e69: Bootstrap starting.
I20260924 19:04:59.737053 20711 log.cc:824] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 1443227823bb45f9afd194a5d0d303b4: Log is configured to *not* fsync() on all Append() calls
I20260924 19:04:59.738662 20710 tablet_bootstrap.cc:492] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44: No bootstrap required, opened a new log
I20260924 19:04:59.738940 20710 ts_tablet_manager.cc:1397] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260924 19:04:59.739917 20712 tablet_bootstrap.cc:654] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 469fb42bacce4f3b9265b7e9f4c67e69: Neither blocks nor log segments found. Creating new log.
I20260924 19:04:59.740161 20711 tablet_bootstrap.cc:492] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 1443227823bb45f9afd194a5d0d303b4: No bootstrap required, opened a new log
I20260924 19:04:59.740434 20711 ts_tablet_manager.cc:1397] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 1443227823bb45f9afd194a5d0d303b4: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260924 19:04:59.741075 20712 log.cc:824] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 469fb42bacce4f3b9265b7e9f4c67e69: Log is configured to *not* fsync() on all Append() calls
I20260924 19:04:59.748416 20710 raft_consensus.cc:348] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "469fb42bacce4f3b9265b7e9f4c67e69" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46583 } } peers { permanent_uuid: "d1096652909d44c1aa25ee0d4e060c44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44233 } } peers { permanent_uuid: "1443227823bb45f9afd194a5d0d303b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40927 } }
I20260924 19:04:59.749281 20712 tablet_bootstrap.cc:492] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 469fb42bacce4f3b9265b7e9f4c67e69: No bootstrap required, opened a new log
I20260924 19:04:59.749601 20712 ts_tablet_manager.cc:1397] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 469fb42bacce4f3b9265b7e9f4c67e69: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.005s
I20260924 19:04:59.750204 20710 raft_consensus.cc:374] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:04:59.750383 20710 raft_consensus.cc:729] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1096652909d44c1aa25ee0d4e060c44, State: Initialized, Role: FOLLOWER
I20260924 19:04:59.751176 20710 consensus_queue.cc:260] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "469fb42bacce4f3b9265b7e9f4c67e69" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46583 } } peers { permanent_uuid: "d1096652909d44c1aa25ee0d4e060c44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44233 } } peers { permanent_uuid: "1443227823bb45f9afd194a5d0d303b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40927 } }
I20260924 19:04:59.752381 20711 raft_consensus.cc:348] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 1443227823bb45f9afd194a5d0d303b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "469fb42bacce4f3b9265b7e9f4c67e69" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46583 } } peers { permanent_uuid: "d1096652909d44c1aa25ee0d4e060c44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44233 } } peers { permanent_uuid: "1443227823bb45f9afd194a5d0d303b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40927 } }
I20260924 19:04:59.752815 20711 raft_consensus.cc:374] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 1443227823bb45f9afd194a5d0d303b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:04:59.752923 20711 raft_consensus.cc:729] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 1443227823bb45f9afd194a5d0d303b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1443227823bb45f9afd194a5d0d303b4, State: Initialized, Role: FOLLOWER
I20260924 19:04:59.753578 20711 consensus_queue.cc:260] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 1443227823bb45f9afd194a5d0d303b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "469fb42bacce4f3b9265b7e9f4c67e69" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46583 } } peers { permanent_uuid: "d1096652909d44c1aa25ee0d4e060c44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44233 } } peers { permanent_uuid: "1443227823bb45f9afd194a5d0d303b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40927 } }
I20260924 19:04:59.758553 20711 ts_tablet_manager.cc:1428] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 1443227823bb45f9afd194a5d0d303b4: Time spent starting tablet: real 0.018s user 0.015s sys 0.003s
I20260924 19:04:59.758919 20562 heartbeater.cc:503] Master 127.0.97.62:33605 was elected leader, sending a full tablet report...
I20260924 19:04:59.761888 20710 ts_tablet_manager.cc:1428] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44: Time spent starting tablet: real 0.023s user 0.002s sys 0.020s
I20260924 19:04:59.762712 20431 heartbeater.cc:503] Master 127.0.97.62:33605 was elected leader, sending a full tablet report...
I20260924 19:04:59.763180 20712 raft_consensus.cc:348] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 469fb42bacce4f3b9265b7e9f4c67e69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "469fb42bacce4f3b9265b7e9f4c67e69" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46583 } } peers { permanent_uuid: "d1096652909d44c1aa25ee0d4e060c44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44233 } } peers { permanent_uuid: "1443227823bb45f9afd194a5d0d303b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40927 } }
I20260924 19:04:59.763594 20712 raft_consensus.cc:374] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 469fb42bacce4f3b9265b7e9f4c67e69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:04:59.763693 20712 raft_consensus.cc:729] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 469fb42bacce4f3b9265b7e9f4c67e69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 469fb42bacce4f3b9265b7e9f4c67e69, State: Initialized, Role: FOLLOWER
I20260924 19:04:59.764304 20712 consensus_queue.cc:260] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 469fb42bacce4f3b9265b7e9f4c67e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "469fb42bacce4f3b9265b7e9f4c67e69" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46583 } } peers { permanent_uuid: "d1096652909d44c1aa25ee0d4e060c44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44233 } } peers { permanent_uuid: "1443227823bb45f9afd194a5d0d303b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40927 } }
I20260924 19:04:59.766562 20692 heartbeater.cc:503] Master 127.0.97.62:33605 was elected leader, sending a full tablet report...
I20260924 19:04:59.767179 20712 ts_tablet_manager.cc:1428] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 469fb42bacce4f3b9265b7e9f4c67e69: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260924 19:04:59.832535 20716 raft_consensus.cc:482] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:04:59.832862 20716 raft_consensus.cc:504] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "469fb42bacce4f3b9265b7e9f4c67e69" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46583 } } peers { permanent_uuid: "d1096652909d44c1aa25ee0d4e060c44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44233 } } peers { permanent_uuid: "1443227823bb45f9afd194a5d0d303b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40927 } }
I20260924 19:04:59.834649 20716 leader_election.cc:288] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 469fb42bacce4f3b9265b7e9f4c67e69 (127.0.97.3:46583), 1443227823bb45f9afd194a5d0d303b4 (127.0.97.2:40927)
I20260924 19:04:59.843458 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:59.835157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41455 (local address 127.0.97.2:40927)
0924 19:04:59.835407 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:59.835416 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:59.836150 (+ 734us) server_negotiation.cc:407] Connection header received
0924 19:04:59.836293 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:59.836298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:59.836387 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:59.836477 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:59.837155 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.838216 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.841181 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.842435 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.842563 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:59.842656 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:59.843082 (+ 426us) server_negotiation.cc:299] Negotiation successful
0924 19:04:59.843205 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 19:04:59.843457 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:59.835140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42551 (local address 127.0.97.3:46583)
0924 19:04:59.835399 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:59.835408 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:59.835873 (+ 465us) server_negotiation.cc:407] Connection header received
0924 19:04:59.836101 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:59.836111 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:59.836222 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:59.836308 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:04:59.837035 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.838216 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.841185 (+ 2969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.842445 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.842563 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:59.842659 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:59.843081 (+ 422us) server_negotiation.cc:299] Negotiation successful
0924 19:04:59.843205 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 19:04:59.843611 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:59.834967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40927 (local address 127.0.97.1:41455)
0924 19:04:59.836057 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:59.836071 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:59.836176 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:59.836497 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:59.836503 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:59.836539 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:59.836982 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:59.836994 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.838356 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.838361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:59.840964 (+ 2603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:59.840980 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.842672 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.842680 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:59.842876 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:59.842951 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 19:04:59.843041 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 19:04:59.843201 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":984,"mutex_wait_us":199,"thread_start_us":763,"threads_started":1}
I20260924 19:04:59.843904 20720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:59.834954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46583 (local address 127.0.97.1:42551)
0924 19:04:59.835769 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0924 19:04:59.835784 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:04:59.835955 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:04:59.836329 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:04:59.836332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:04:59.836369 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:04:59.836878 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:59.836892 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.838357 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.838361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:59.840965 (+ 2604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:04:59.840980 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.842688 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.842694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:04:59.842881 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:59.842946 (+ 65us) client_negotiation.cc:769] Sending connection context
0924 19:04:59.843041 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 19:04:59.843201 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":171,"spinlock_wait_cycles":24320,"thread_start_us":457,"threads_started":1}
I20260924 19:04:59.844539 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4f405eb3acf4ee5ab0c48c65fb8c69a" candidate_uuid: "d1096652909d44c1aa25ee0d4e060c44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1443227823bb45f9afd194a5d0d303b4" is_pre_election: true
I20260924 19:04:59.844892 20647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4f405eb3acf4ee5ab0c48c65fb8c69a" candidate_uuid: "d1096652909d44c1aa25ee0d4e060c44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "469fb42bacce4f3b9265b7e9f4c67e69" is_pre_election: true
I20260924 19:04:59.845028 20517 raft_consensus.cc:2454] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 1443227823bb45f9afd194a5d0d303b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1096652909d44c1aa25ee0d4e060c44 in term 0.
I20260924 19:04:59.845333 20647 raft_consensus.cc:2454] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 469fb42bacce4f3b9265b7e9f4c67e69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1096652909d44c1aa25ee0d4e060c44 in term 0.
I20260924 19:04:59.845698 20319 leader_election.cc:302] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1443227823bb45f9afd194a5d0d303b4, d1096652909d44c1aa25ee0d4e060c44; no voters:
I20260924 19:04:59.846086 20716 raft_consensus.cc:2781] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:04:59.846242 20716 raft_consensus.cc:482] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:04:59.846350 20716 raft_consensus.cc:3037] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:04:59.848532 20716 raft_consensus.cc:504] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "469fb42bacce4f3b9265b7e9f4c67e69" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46583 } } peers { permanent_uuid: "d1096652909d44c1aa25ee0d4e060c44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44233 } } peers { permanent_uuid: "1443227823bb45f9afd194a5d0d303b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40927 } }
I20260924 19:04:59.849205 20716 leader_election.cc:288] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44 [CANDIDATE]: Term 1 election: Requested vote from peers 469fb42bacce4f3b9265b7e9f4c67e69 (127.0.97.3:46583), 1443227823bb45f9afd194a5d0d303b4 (127.0.97.2:40927)
I20260924 19:04:59.849598 20647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4f405eb3acf4ee5ab0c48c65fb8c69a" candidate_uuid: "d1096652909d44c1aa25ee0d4e060c44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "469fb42bacce4f3b9265b7e9f4c67e69"
I20260924 19:04:59.849694 20517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4f405eb3acf4ee5ab0c48c65fb8c69a" candidate_uuid: "d1096652909d44c1aa25ee0d4e060c44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1443227823bb45f9afd194a5d0d303b4"
I20260924 19:04:59.849782 20647 raft_consensus.cc:3037] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 469fb42bacce4f3b9265b7e9f4c67e69 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:04:59.849921 20517 raft_consensus.cc:3037] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 1443227823bb45f9afd194a5d0d303b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:04:59.852011 20647 raft_consensus.cc:2454] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 469fb42bacce4f3b9265b7e9f4c67e69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1096652909d44c1aa25ee0d4e060c44 in term 1.
I20260924 19:04:59.852240 20517 raft_consensus.cc:2454] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 1443227823bb45f9afd194a5d0d303b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1096652909d44c1aa25ee0d4e060c44 in term 1.
I20260924 19:04:59.852416 20321 leader_election.cc:302] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 469fb42bacce4f3b9265b7e9f4c67e69, d1096652909d44c1aa25ee0d4e060c44; no voters:
I20260924 19:04:59.852722 20716 raft_consensus.cc:2781] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:04:59.853277 20716 raft_consensus.cc:686] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44 [term 1 LEADER]: Becoming Leader. State: Replica: d1096652909d44c1aa25ee0d4e060c44, State: Running, Role: LEADER
I20260924 19:04:59.853790 20716 consensus_queue.cc:237] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "469fb42bacce4f3b9265b7e9f4c67e69" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46583 } } peers { permanent_uuid: "d1096652909d44c1aa25ee0d4e060c44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44233 } } peers { permanent_uuid: "1443227823bb45f9afd194a5d0d303b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40927 } }
I20260924 19:04:59.858098 20244 catalog_manager.cc:5495] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1096652909d44c1aa25ee0d4e060c44 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "d1096652909d44c1aa25ee0d4e060c44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "469fb42bacce4f3b9265b7e9f4c67e69" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1096652909d44c1aa25ee0d4e060c44" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1443227823bb45f9afd194a5d0d303b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40927 } health_report { overall_health: UNKNOWN } } }
W20260924 19:04:59.888429 20693 tablet.cc:2369] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 469fb42bacce4f3b9265b7e9f4c67e69: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:04:59.903373 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:59.898667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37200 (local address 127.0.97.1:44233)
0924 19:04:59.898838 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:59.898842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:59.898859 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:04:59.898965 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:59.898969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:59.899060 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:59.899186 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:04:59.899467 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.900171 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.901471 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.901905 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.901997 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:59.902427 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:04:59.902910 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:04:59.903015 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:59.903101 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 19:04:59.903172 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 19:04:59.911319 20517 raft_consensus.cc:1261] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 1443227823bb45f9afd194a5d0d303b4 [term 1 FOLLOWER]: Refusing update from remote peer d1096652909d44c1aa25ee0d4e060c44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:04:59.911319 20647 raft_consensus.cc:1261] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 469fb42bacce4f3b9265b7e9f4c67e69 [term 1 FOLLOWER]: Refusing update from remote peer d1096652909d44c1aa25ee0d4e060c44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:04:59.912308 20721 consensus_queue.cc:1035] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "469fb42bacce4f3b9265b7e9f4c67e69" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:04:59.912674 20716 consensus_queue.cc:1035] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1443227823bb45f9afd194a5d0d303b4" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:04:59.937649 20244 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:04:59.937916 20244 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:04:59.941068 20244 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8504e3bcaaaa41d1aa1120be61261835: Sending DeleteTablet for 3 replicas of tablet d4f405eb3acf4ee5ab0c48c65fb8c69a
I20260924 19:04:59.942157 20627 tablet_service.cc:1514] Processing DeleteTablet for tablet d4f405eb3acf4ee5ab0c48c65fb8c69a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:04:59 UTC) from {username='slave'} at 127.0.0.1:36678
I20260924 19:04:59.942324 20366 tablet_service.cc:1514] Processing DeleteTablet for tablet d4f405eb3acf4ee5ab0c48c65fb8c69a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:04:59 UTC) from {username='slave'} at 127.0.0.1:37190
I20260924 19:04:59.942323 20497 tablet_service.cc:1514] Processing DeleteTablet for tablet d4f405eb3acf4ee5ab0c48c65fb8c69a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:04:59 UTC) from {username='slave'} at 127.0.0.1:60090
I20260924 19:04:59.944350 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 20304
I20260924 19:04:59.958299 20735 tablet_replica.cc:330] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44: stopping tablet replica
I20260924 19:04:59.958843 20735 raft_consensus.cc:2229] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:04:59.959713 20735 raft_consensus.cc:2258] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:04:59.961719 20736 tablet_replica.cc:330] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 1443227823bb45f9afd194a5d0d303b4: stopping tablet replica
I20260924 19:04:59.961959 20734 tablet_replica.cc:330] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 469fb42bacce4f3b9265b7e9f4c67e69: stopping tablet replica
I20260924 19:04:59.962137 20736 raft_consensus.cc:2229] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 1443227823bb45f9afd194a5d0d303b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:04:59.962430 20734 raft_consensus.cc:2229] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 469fb42bacce4f3b9265b7e9f4c67e69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:04:59.962646 20736 raft_consensus.cc:2258] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 1443227823bb45f9afd194a5d0d303b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:04:59.962800 20735 ts_tablet_manager.cc:1905] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:04:59.962997 20734 raft_consensus.cc:2258] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 469fb42bacce4f3b9265b7e9f4c67e69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:04:59.965106 20736 ts_tablet_manager.cc:1905] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 1443227823bb45f9afd194a5d0d303b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:04:59.969072 20707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:04:59.946082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37212 (local address 127.0.97.1:44233)
0924 19:04:59.946335 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 19:04:59.946344 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:04:59.960923 (+ 14579us) server_negotiation.cc:407] Connection header received
0924 19:04:59.961170 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:04:59.961175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:04:59.961273 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:04:59.961378 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:04:59.962054 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.963137 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.965064 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:04:59.965766 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:04:59.965860 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:04:59.967289 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:04:59.967300 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:04:59.967318 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:04:59.968144 (+ 826us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:04:59.968208 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:04:59.968230 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:04:59.968316 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 19:04:59.968572 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 19:04:59.968683 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 19:04:59.970260 20426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:04:59.970840 20734 ts_tablet_manager.cc:1905] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 469fb42bacce4f3b9265b7e9f4c67e69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:04:59.973546 20736 ts_tablet_manager.cc:1918] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 1443227823bb45f9afd194a5d0d303b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:04:59.973726 20736 log.cc:1196] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 1443227823bb45f9afd194a5d0d303b4: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-1/wal/wals/d4f405eb3acf4ee5ab0c48c65fb8c69a
I20260924 19:04:59.974283 20736 ts_tablet_manager.cc:1939] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 1443227823bb45f9afd194a5d0d303b4: Deleting consensus metadata
I20260924 19:04:59.976221 20229 catalog_manager.cc:4841] TS 1443227823bb45f9afd194a5d0d303b4 (127.0.97.2:40927): tablet d4f405eb3acf4ee5ab0c48c65fb8c69a (table test-table [id=a753ecc0ca1a484a8f80380847d7fad5]) successfully deleted
I20260924 19:04:59.977253 20734 ts_tablet_manager.cc:1918] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 469fb42bacce4f3b9265b7e9f4c67e69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:04:59.977427 20734 log.cc:1196] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 469fb42bacce4f3b9265b7e9f4c67e69: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-2/wal/wals/d4f405eb3acf4ee5ab0c48c65fb8c69a
I20260924 19:04:59.977903 20734 ts_tablet_manager.cc:1939] T d4f405eb3acf4ee5ab0c48c65fb8c69a P 469fb42bacce4f3b9265b7e9f4c67e69: Deleting consensus metadata
I20260924 19:04:59.979481 20231 catalog_manager.cc:4841] TS 469fb42bacce4f3b9265b7e9f4c67e69 (127.0.97.3:46583): tablet d4f405eb3acf4ee5ab0c48c65fb8c69a (table test-table [id=a753ecc0ca1a484a8f80380847d7fad5]) successfully deleted
I20260924 19:05:00.066627 20735 ts_tablet_manager.cc:1918] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:05:00.066910 20735 log.cc:1196] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44: Deleting WAL directory at /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276386554690-388-0/minicluster-data/ts-0/wal/wals/d4f405eb3acf4ee5ab0c48c65fb8c69a
I20260924 19:05:00.067488 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 20304
I20260924 19:05:00.067597 20735 ts_tablet_manager.cc:1939] T d4f405eb3acf4ee5ab0c48c65fb8c69a P d1096652909d44c1aa25ee0d4e060c44: Deleting consensus metadata
I20260924 19:05:00.086277 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 20435
W20260924 19:05:00.086241 20229 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44233 (error 108)
I20260924 19:05:00.088138 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:00.087675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44233 (local address 127.0.0.1:37224)
0924 19:05:00.087853 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:00.087962 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 19:05:00.088377 20229 catalog_manager.cc:4584] TS d1096652909d44c1aa25ee0d4e060c44 (127.0.97.1:44233): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d4f405eb3acf4ee5ab0c48c65fb8c69a: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44233: connect: Connection refused (error 111)
I20260924 19:05:00.093302 20708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:00.087148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60092 (local address 127.0.97.2:40927)
0924 19:05:00.087356 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:00.087363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:00.087636 (+ 273us) server_negotiation.cc:407] Connection header received
0924 19:05:00.087881 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:00.087885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:00.087968 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:00.088059 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:00.088531 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:00.089284 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:00.090510 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:00.090878 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:00.090977 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:00.091956 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:00.091967 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:00.091980 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:05:00.092550 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:00.092599 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:05:00.092618 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:00.092700 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:00.092803 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 19:05:00.092943 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 19:05:00.094132 20557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:05:00.141896 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:00.141423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44233 (local address 127.0.0.1:37228)
0924 19:05:00.141657 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:00.141742 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260924 19:05:00.142879 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:00.142538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44233 (local address 127.0.0.1:37230)
0924 19:05:00.142693 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:00.142759 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 19:05:00.181893 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 20435
I20260924 19:05:00.198612 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 20565
I20260924 19:05:00.203512 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:00.203048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44233 (local address 127.0.0.1:37238)
0924 19:05:00.203235 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:00.203309 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 19:05:00.204640 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:00.204187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44233 (local address 127.0.0.1:37248)
0924 19:05:00.204380 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:00.204486 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 19:05:00.211582 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:00.199427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36692 (local address 127.0.97.3:46583)
0924 19:05:00.199647 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:00.199653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:00.199673 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:05:00.199736 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:00.199742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:00.199845 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:00.199945 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:00.200301 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:00.201444 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:00.202695 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:00.203341 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:00.203461 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:00.210436 (+ 6975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:00.210445 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:00.210461 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:05:00.211010 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:00.211057 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:05:00.211077 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:00.211167 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:00.211272 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 19:05:00.211363 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:05:00.212230 20687 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:05:00.285403 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:00.284853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44233 (local address 127.0.0.1:37258)
0924 19:05:00.285101 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:00.285217 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 19:05:00.286545 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:00.286101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44233 (local address 127.0.0.1:37274)
0924 19:05:00.286336 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:00.286418 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260924 19:05:00.300583 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 20565
I20260924 19:05:00.316785 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 20214
I20260924 19:05:00.317870 20275 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:05:00.426148 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 20214
2026-09-24T19:05:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2730 ms)
[----------] 6 tests from EncryptionPolicyTest (20866 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:05:00 dist-test-slave-9fn1 krb5kdc[20745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:05:00 dist-test-slave-9fn1 krb5kdc[20745](info): set up 2 sockets
Sep 24 19:05:00 dist-test-slave-9fn1 krb5kdc[20745](info): commencing operation
krb5kdc: starting...
W20260924 19:05:01.601902 388 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:05:01 dist-test-slave-9fn1 krb5kdc[20745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:05:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:05:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:05:01.852403 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43431
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:02.181280 20758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:02.181566 20758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:02.181659 20758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:02.188917 20758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:05:02.189020 20758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:02.189065 20758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:02.189126 20758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:05:02.189165 20758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:05:02.199091 20758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43431
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:02.200248 20758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:02.201602 20758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:02.213099 20763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:02.213136 20764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:02.213349 20766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:02.213970 20758 server_base.cc:1034] running on GCE node
I20260924 19:05:02.214699 20758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:02.216019 20758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:02.217274 20758 hybrid_clock.cc:648] HybridClock initialized: now 1790276702217229 us; error 35 us; skew 500 ppm
I20260924 19:05:02.217717 20758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:02 dist-test-slave-9fn1 krb5kdc[20745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:02.220765 20758 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:05:02.222290 20758 webserver.cc:460] Webserver started at http://127.0.0.1:41439/ using document root <none> and password file <none>
I20260924 19:05:02.222960 20758 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:02.223099 20758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:02.223407 20758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:02.226385 20758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "ae5d611a1e7143ecb349c61d90c9e15a"
format_stamp: "Formatted at 2026-09-24 19:05:02 on dist-test-slave-9fn1"
server_key: "6b0bb31fed812d9653746790a58d663d"
server_key_iv: "a13aabb05389a26469c13308d138d6de"
server_key_version: "encryptionkey@0"
I20260924 19:05:02.227138 20758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "ae5d611a1e7143ecb349c61d90c9e15a"
format_stamp: "Formatted at 2026-09-24 19:05:02 on dist-test-slave-9fn1"
server_key: "6b0bb31fed812d9653746790a58d663d"
server_key_iv: "a13aabb05389a26469c13308d138d6de"
server_key_version: "encryptionkey@0"
I20260924 19:05:02.232007 20758 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:05:02.235309 20773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:02.236497 20758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 19:05:02.236658 20758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "ae5d611a1e7143ecb349c61d90c9e15a"
format_stamp: "Formatted at 2026-09-24 19:05:02 on dist-test-slave-9fn1"
server_key: "6b0bb31fed812d9653746790a58d663d"
server_key_iv: "a13aabb05389a26469c13308d138d6de"
server_key_version: "encryptionkey@0"
I20260924 19:05:02.236856 20758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:02.260591 20758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:02.264371 20758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:02.264649 20758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:02.282663 20758 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43431
I20260924 19:05:02.282698 20824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43431 every 8 connection(s)
I20260924 19:05:02.283993 20758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:05:02.285681 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 20758
I20260924 19:05:02.285885 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:05:02.286650 388 external_mini_cluster.cc:1428] Setting key 41219935c7ab07bc795e4dba8fa74c17
I20260924 19:05:02.288631 20825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 19:05:02 dist-test-slave-9fn1 krb5kdc[20745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 19:05:02.302467 20825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae5d611a1e7143ecb349c61d90c9e15a: Bootstrap starting.
I20260924 19:05:02.306689 20825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae5d611a1e7143ecb349c61d90c9e15a: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:02.307878 20825 log.cc:824] T 00000000000000000000000000000000 P ae5d611a1e7143ecb349c61d90c9e15a: Log is configured to *not* fsync() on all Append() calls
I20260924 19:05:02.311144 20828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:02.288529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33593 (local address 127.0.0.1:43431)
0924 19:05:02.289308 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:02.289319 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:02.289357 (+ 38us) server_negotiation.cc:407] Connection header received
0924 19:05:02.291014 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:02.291034 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:02.291403 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:02.291737 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:02.293615 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:02.294970 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:02.296749 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:02.297618 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:02.297741 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:02.302649 (+ 4908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:02.302660 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:02.302680 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:02.302722 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:02.306501 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:02.307346 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:02.307354 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:02.307362 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:02.307596 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:02.308013 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:02.308018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:02.308020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:02.308695 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:02.308932 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:02.309601 (+ 669us) server_negotiation.cc:299] Negotiation successful
0924 19:05:02.309918 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":372,"threads_started":1}
I20260924 19:05:02.311280 20825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae5d611a1e7143ecb349c61d90c9e15a: No bootstrap required, opened a new log
I20260924 19:05:02.315033 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:05:02.320068 20825 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae5d611a1e7143ecb349c61d90c9e15a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5d611a1e7143ecb349c61d90c9e15a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43431 } }
I20260924 19:05:02.320608 20825 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae5d611a1e7143ecb349c61d90c9e15a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:05:02.320722 20825 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae5d611a1e7143ecb349c61d90c9e15a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5d611a1e7143ecb349c61d90c9e15a, State: Initialized, Role: FOLLOWER
I20260924 19:05:02.321264 20825 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae5d611a1e7143ecb349c61d90c9e15a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5d611a1e7143ecb349c61d90c9e15a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43431 } }
I20260924 19:05:02.321527 20825 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae5d611a1e7143ecb349c61d90c9e15a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:05:02.321679 20825 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae5d611a1e7143ecb349c61d90c9e15a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:05:02.321841 20825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae5d611a1e7143ecb349c61d90c9e15a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:02.324087 20825 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae5d611a1e7143ecb349c61d90c9e15a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5d611a1e7143ecb349c61d90c9e15a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43431 } }
I20260924 19:05:02.324581 20825 leader_election.cc:302] T 00000000000000000000000000000000 P ae5d611a1e7143ecb349c61d90c9e15a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae5d611a1e7143ecb349c61d90c9e15a; no voters:
I20260924 19:05:02.325237 20825 leader_election.cc:288] T 00000000000000000000000000000000 P ae5d611a1e7143ecb349c61d90c9e15a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:05:02.325361 20835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae5d611a1e7143ecb349c61d90c9e15a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:05:02.331802 20825 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae5d611a1e7143ecb349c61d90c9e15a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:05:02.332242 20835 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae5d611a1e7143ecb349c61d90c9e15a [term 1 LEADER]: Becoming Leader. State: Replica: ae5d611a1e7143ecb349c61d90c9e15a, State: Running, Role: LEADER
I20260924 19:05:02.332928 20835 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae5d611a1e7143ecb349c61d90c9e15a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5d611a1e7143ecb349c61d90c9e15a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43431 } }
I20260924 19:05:02.333727 20828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:02.320049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40567 (local address 127.0.0.1:43431)
0924 19:05:02.320221 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:02.320229 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:02.320671 (+ 442us) server_negotiation.cc:407] Connection header received
0924 19:05:02.320746 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:02.320751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:02.320836 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:02.320920 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:02.322696 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:02.323527 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:02.324916 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:02.325489 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:02.325667 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:02.328036 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:02.328043 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:02.328047 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:02.328084 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:02.330785 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:02.331625 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:02.331630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:02.331634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:02.331774 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:02.332244 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:02.332280 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:02.332283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:02.332608 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:02.332949 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:02.333237 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 19:05:02.333416 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 19:05:02.339295 20836 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae5d611a1e7143ecb349c61d90c9e15a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae5d611a1e7143ecb349c61d90c9e15a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5d611a1e7143ecb349c61d90c9e15a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43431 } } }
I20260924 19:05:02.339331 20837 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae5d611a1e7143ecb349c61d90c9e15a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae5d611a1e7143ecb349c61d90c9e15a. Latest consensus state: current_term: 1 leader_uuid: "ae5d611a1e7143ecb349c61d90c9e15a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae5d611a1e7143ecb349c61d90c9e15a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43431 } } }
I20260924 19:05:02.339888 20837 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae5d611a1e7143ecb349c61d90c9e15a [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:02.339862 20836 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae5d611a1e7143ecb349c61d90c9e15a [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:02.342667 20845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:05:02.346834 20845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:05:02.354808 20845 catalog_manager.cc:1347] Generated new cluster ID: c4e22e74193c450c84a1b7d739dbe7dd
I20260924 19:05:02.354918 20845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:05:02.374181 20845 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:05:02.375483 20845 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:05:02.382300 20845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae5d611a1e7143ecb349c61d90c9e15a: Generated new TSK 0
I20260924 19:05:02.383072 20845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:05:02.393195 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 20758
I20260924 19:05:02.395323 20820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:05:02.483642 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 20758
W20260924 19:05:02.500839 20743 connection.cc:535] client connection to 127.0.0.1:43431 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43431): Connection reset by peer (error 104)
2026-09-24T19:05:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2052 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:05:02 dist-test-slave-9fn1 krb5kdc[20857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:05:02 dist-test-slave-9fn1 krb5kdc[20857](info): set up 2 sockets
Sep 24 19:05:02 dist-test-slave-9fn1 krb5kdc[20857](info): commencing operation
krb5kdc: starting...
W20260924 19:05:03.585721 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:05:03 dist-test-slave-9fn1 krb5kdc[20857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:05:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:05:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:05:03.821246 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45571
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:04.145027 20870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:04.145370 20870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:04.145555 20870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:04.152711 20870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:05:04.152860 20870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:04.152954 20870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:04.153071 20870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:05:04.153147 20870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:05:04.162904 20870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45571
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:04.164218 20870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:04.165483 20870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:04.176931 20875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:04.176986 20876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:04.177009 20878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:04.178328 20870 server_base.cc:1034] running on GCE node
I20260924 19:05:04.179069 20870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:04.180383 20870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:04.181597 20870 hybrid_clock.cc:648] HybridClock initialized: now 1790276704181566 us; error 58 us; skew 500 ppm
I20260924 19:05:04.181924 20870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:04 dist-test-slave-9fn1 krb5kdc[20857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:04.185043 20870 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:05:04.186467 20870 webserver.cc:460] Webserver started at http://127.0.0.1:45329/ using document root <none> and password file <none>
I20260924 19:05:04.187078 20870 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:04.187173 20870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:04.187417 20870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:04.190274 20870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "d6e5afccd0624163bca03c1b732c75d9"
format_stamp: "Formatted at 2026-09-24 19:05:04 on dist-test-slave-9fn1"
server_key: "ec6a49a430dd54e0cacda6fc6d234620"
server_key_iv: "847714723297293605f6300a3dbbf534"
server_key_version: "encryptionkey@0"
I20260924 19:05:04.190963 20870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "d6e5afccd0624163bca03c1b732c75d9"
format_stamp: "Formatted at 2026-09-24 19:05:04 on dist-test-slave-9fn1"
server_key: "ec6a49a430dd54e0cacda6fc6d234620"
server_key_iv: "847714723297293605f6300a3dbbf534"
server_key_version: "encryptionkey@0"
I20260924 19:05:04.195849 20870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:05:04.199043 20885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:04.200295 20870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:05:04.200465 20870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "d6e5afccd0624163bca03c1b732c75d9"
format_stamp: "Formatted at 2026-09-24 19:05:04 on dist-test-slave-9fn1"
server_key: "ec6a49a430dd54e0cacda6fc6d234620"
server_key_iv: "847714723297293605f6300a3dbbf534"
server_key_version: "encryptionkey@0"
I20260924 19:05:04.200650 20870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:04.236896 20870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:04.240914 20870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:04.241204 20870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:04.259338 20870 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45571
I20260924 19:05:04.259368 20936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45571 every 8 connection(s)
I20260924 19:05:04.260695 20870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:05:04.263625 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 20870
I20260924 19:05:04.263801 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:05:04.264194 388 external_mini_cluster.cc:1428] Setting key c640638e1af77ecae0e78cd647096c0a
I20260924 19:05:04.264475 20937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:05:04.279417 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:04.265956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56999 (local address 127.0.0.1:45571)
0924 19:05:04.266711 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:04.266726 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:04.266774 (+ 48us) server_negotiation.cc:407] Connection header received
0924 19:05:04.269161 (+ 2387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:04.269189 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:04.269740 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:04.270172 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:04.270677 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:04.272701 (+ 2024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:04.274152 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:04.275233 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:04.275450 (+ 217us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:04.276710 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:04.276721 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:04.276738 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:05:04.277470 (+ 732us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:04.277573 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:05:04.277602 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:04.277753 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:04.277988 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 19:05:04.278159 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":316,"threads_started":1}
I20260924 19:05:04.279510 20937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6e5afccd0624163bca03c1b732c75d9: Bootstrap starting.
I20260924 19:05:04.282562 20937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6e5afccd0624163bca03c1b732c75d9: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:04.282968 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:05:04.283742 20937 log.cc:824] T 00000000000000000000000000000000 P d6e5afccd0624163bca03c1b732c75d9: Log is configured to *not* fsync() on all Append() calls
I20260924 19:05:04.286739 20937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6e5afccd0624163bca03c1b732c75d9: No bootstrap required, opened a new log
I20260924 19:05:04.294045 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:04.287625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42245 (local address 127.0.0.1:45571)
0924 19:05:04.287818 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:04.287823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:04.288139 (+ 316us) server_negotiation.cc:407] Connection header received
0924 19:05:04.288274 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:04.288279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:04.288351 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:04.288459 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:04.288870 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:04.290035 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:04.291323 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:04.292020 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:04.292162 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:04.293234 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:04.293241 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:04.293246 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:05:04.293268 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:04.293314 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:05:04.293332 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:04.293456 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:04.293659 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 19:05:04.293718 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 19:05:04.296139 20937 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6e5afccd0624163bca03c1b732c75d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e5afccd0624163bca03c1b732c75d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45571 } }
I20260924 19:05:04.296469 20937 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6e5afccd0624163bca03c1b732c75d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:05:04.296543 20937 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6e5afccd0624163bca03c1b732c75d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e5afccd0624163bca03c1b732c75d9, State: Initialized, Role: FOLLOWER
I20260924 19:05:04.297015 20937 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6e5afccd0624163bca03c1b732c75d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e5afccd0624163bca03c1b732c75d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45571 } }
I20260924 19:05:04.297209 20937 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6e5afccd0624163bca03c1b732c75d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:05:04.297303 20937 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6e5afccd0624163bca03c1b732c75d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:05:04.297423 20937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6e5afccd0624163bca03c1b732c75d9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:04.299513 20937 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6e5afccd0624163bca03c1b732c75d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e5afccd0624163bca03c1b732c75d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45571 } }
I20260924 19:05:04.299932 20937 leader_election.cc:302] T 00000000000000000000000000000000 P d6e5afccd0624163bca03c1b732c75d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6e5afccd0624163bca03c1b732c75d9; no voters:
I20260924 19:05:04.300549 20937 leader_election.cc:288] T 00000000000000000000000000000000 P d6e5afccd0624163bca03c1b732c75d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:05:04.300660 20947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6e5afccd0624163bca03c1b732c75d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:05:04.301409 20947 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6e5afccd0624163bca03c1b732c75d9 [term 1 LEADER]: Becoming Leader. State: Replica: d6e5afccd0624163bca03c1b732c75d9, State: Running, Role: LEADER
I20260924 19:05:04.302022 20947 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6e5afccd0624163bca03c1b732c75d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e5afccd0624163bca03c1b732c75d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45571 } }
I20260924 19:05:04.302793 20937 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6e5afccd0624163bca03c1b732c75d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:05:04.304556 20948 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6e5afccd0624163bca03c1b732c75d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6e5afccd0624163bca03c1b732c75d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e5afccd0624163bca03c1b732c75d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45571 } } }
I20260924 19:05:04.304893 20948 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6e5afccd0624163bca03c1b732c75d9 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:04.305773 20953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:05:04.305593 20949 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6e5afccd0624163bca03c1b732c75d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6e5afccd0624163bca03c1b732c75d9. Latest consensus state: current_term: 1 leader_uuid: "d6e5afccd0624163bca03c1b732c75d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e5afccd0624163bca03c1b732c75d9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45571 } } }
I20260924 19:05:04.305950 20949 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6e5afccd0624163bca03c1b732c75d9 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:04.311244 20953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:05:04.319973 20953 catalog_manager.cc:1347] Generated new cluster ID: 8dec4e2b00a246a3ad7f53bdbfca3a7a
I20260924 19:05:04.320088 20953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:05:04.350800 20953 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:05:04.352173 20953 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:05:04.364651 20953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6e5afccd0624163bca03c1b732c75d9: Generated new TSK 0
I20260924 19:05:04.365578 20953 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:05:04.381568 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 20870
W20260924 19:05:04.382707 20932 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56999
W20260924 19:05:04.383417 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 20870: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:05:04.383600 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 20870
W20260924 19:05:04.400193 20855 connection.cc:535] client connection to 127.0.0.1:45571 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45571): Connection reset by peer (error 104)
2026-09-24T19:05:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1903 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:05:04 dist-test-slave-9fn1 krb5kdc[20968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:05:04 dist-test-slave-9fn1 krb5kdc[20968](info): set up 2 sockets
Sep 24 19:05:04 dist-test-slave-9fn1 krb5kdc[20968](info): commencing operation
krb5kdc: starting...
W20260924 19:05:05.469769 388 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:05:05 dist-test-slave-9fn1 krb5kdc[20968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:05:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:05:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:05:05.703900 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42323
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:06.033782 20981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:06.034092 20981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:06.034246 20981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:06.041648 20981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:05:06.041760 20981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:06.041851 20981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:06.041953 20981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:05:06.042032 20981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:05:06.051931 20981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42323
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:06.053207 20981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:06.054533 20981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:06.066612 20986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:06.066571 20987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:06.066651 20989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:06.067569 20981 server_base.cc:1034] running on GCE node
I20260924 19:05:06.068398 20981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:06.069870 20981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:06.071148 20981 hybrid_clock.cc:648] HybridClock initialized: now 1790276706071107 us; error 32 us; skew 500 ppm
I20260924 19:05:06.071542 20981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:06 dist-test-slave-9fn1 krb5kdc[20968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:06.080550 20981 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:05:06.082151 20981 webserver.cc:460] Webserver started at http://127.0.0.1:37581/ using document root <none> and password file <none>
I20260924 19:05:06.082819 20981 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:06.082921 20981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:06.083186 20981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:06.086206 20981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "bde94082738d47eb914b2fa6d7caf86a"
format_stamp: "Formatted at 2026-09-24 19:05:06 on dist-test-slave-9fn1"
server_key: "c939c1fc5e53d179772b361d86e584ba"
server_key_iv: "945a90b4aec8496d8c5ffa93bfabdb40"
server_key_version: "encryptionkey@0"
I20260924 19:05:06.086944 20981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "bde94082738d47eb914b2fa6d7caf86a"
format_stamp: "Formatted at 2026-09-24 19:05:06 on dist-test-slave-9fn1"
server_key: "c939c1fc5e53d179772b361d86e584ba"
server_key_iv: "945a90b4aec8496d8c5ffa93bfabdb40"
server_key_version: "encryptionkey@0"
I20260924 19:05:06.092082 20981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:05:06.095495 20996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:06.096895 20981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:05:06.097074 20981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "bde94082738d47eb914b2fa6d7caf86a"
format_stamp: "Formatted at 2026-09-24 19:05:06 on dist-test-slave-9fn1"
server_key: "c939c1fc5e53d179772b361d86e584ba"
server_key_iv: "945a90b4aec8496d8c5ffa93bfabdb40"
server_key_version: "encryptionkey@0"
I20260924 19:05:06.097288 20981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:06.103086 20981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:06.107385 20981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:06.107622 20981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:06.125942 20981 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42323
I20260924 19:05:06.126019 21047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42323 every 8 connection(s)
I20260924 19:05:06.127415 20981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:05:06.131090 21048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:05:06.136477 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 20981
I20260924 19:05:06.136658 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:05:06.137104 388 external_mini_cluster.cc:1428] Setting key e313ebd67479fb535d011c37accfae90
I20260924 19:05:06.141773 21048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bde94082738d47eb914b2fa6d7caf86a: Bootstrap starting.
I20260924 19:05:06.144708 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:06.138945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40563 (local address 127.0.0.1:42323)
0924 19:05:06.139822 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:06.139833 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:06.139877 (+ 44us) server_negotiation.cc:407] Connection header received
0924 19:05:06.141478 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:06.141570 (+ 92us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:06.141903 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:06.142231 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:06.143233 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:06.143239 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:06.143263 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:05:06.143277 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:06.143339 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:05:06.143358 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:06.143517 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:06.143672 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 19:05:06.143863 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":387,"threads_started":1}
I20260924 19:05:06.145366 21048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bde94082738d47eb914b2fa6d7caf86a: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:06.146528 21048 log.cc:824] T 00000000000000000000000000000000 P bde94082738d47eb914b2fa6d7caf86a: Log is configured to *not* fsync() on all Append() calls
I20260924 19:05:06.148442 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:05:06.149317 21048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bde94082738d47eb914b2fa6d7caf86a: No bootstrap required, opened a new log
I20260924 19:05:06.155171 21051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:06.152835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34057 (local address 127.0.0.1:42323)
0924 19:05:06.153025 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:06.153030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:06.153347 (+ 317us) server_negotiation.cc:407] Connection header received
0924 19:05:06.153470 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:06.153475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:06.153549 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:06.153649 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:06.154527 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:06.154533 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:06.154538 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:05:06.154556 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:06.154602 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:05:06.154620 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:06.154751 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:06.154816 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 19:05:06.154878 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 19:05:06.157814 21048 raft_consensus.cc:348] T 00000000000000000000000000000000 P bde94082738d47eb914b2fa6d7caf86a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde94082738d47eb914b2fa6d7caf86a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42323 } }
I20260924 19:05:06.158207 21048 raft_consensus.cc:374] T 00000000000000000000000000000000 P bde94082738d47eb914b2fa6d7caf86a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:05:06.158326 21048 raft_consensus.cc:729] T 00000000000000000000000000000000 P bde94082738d47eb914b2fa6d7caf86a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bde94082738d47eb914b2fa6d7caf86a, State: Initialized, Role: FOLLOWER
I20260924 19:05:06.158871 21048 consensus_queue.cc:260] T 00000000000000000000000000000000 P bde94082738d47eb914b2fa6d7caf86a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde94082738d47eb914b2fa6d7caf86a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42323 } }
I20260924 19:05:06.159111 21048 raft_consensus.cc:388] T 00000000000000000000000000000000 P bde94082738d47eb914b2fa6d7caf86a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:05:06.159256 21048 raft_consensus.cc:482] T 00000000000000000000000000000000 P bde94082738d47eb914b2fa6d7caf86a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:05:06.159433 21048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bde94082738d47eb914b2fa6d7caf86a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:06.161746 21048 raft_consensus.cc:504] T 00000000000000000000000000000000 P bde94082738d47eb914b2fa6d7caf86a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde94082738d47eb914b2fa6d7caf86a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42323 } }
I20260924 19:05:06.162220 21048 leader_election.cc:302] T 00000000000000000000000000000000 P bde94082738d47eb914b2fa6d7caf86a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bde94082738d47eb914b2fa6d7caf86a; no voters:
I20260924 19:05:06.162762 21048 leader_election.cc:288] T 00000000000000000000000000000000 P bde94082738d47eb914b2fa6d7caf86a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:05:06.162909 21058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bde94082738d47eb914b2fa6d7caf86a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:05:06.163568 21058 raft_consensus.cc:686] T 00000000000000000000000000000000 P bde94082738d47eb914b2fa6d7caf86a [term 1 LEADER]: Becoming Leader. State: Replica: bde94082738d47eb914b2fa6d7caf86a, State: Running, Role: LEADER
I20260924 19:05:06.164036 21058 consensus_queue.cc:237] T 00000000000000000000000000000000 P bde94082738d47eb914b2fa6d7caf86a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde94082738d47eb914b2fa6d7caf86a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42323 } }
I20260924 19:05:06.164724 21048 sys_catalog.cc:564] T 00000000000000000000000000000000 P bde94082738d47eb914b2fa6d7caf86a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:05:06.166970 21059 sys_catalog.cc:455] T 00000000000000000000000000000000 P bde94082738d47eb914b2fa6d7caf86a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bde94082738d47eb914b2fa6d7caf86a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde94082738d47eb914b2fa6d7caf86a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42323 } } }
I20260924 19:05:06.167483 21060 sys_catalog.cc:455] T 00000000000000000000000000000000 P bde94082738d47eb914b2fa6d7caf86a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bde94082738d47eb914b2fa6d7caf86a. Latest consensus state: current_term: 1 leader_uuid: "bde94082738d47eb914b2fa6d7caf86a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde94082738d47eb914b2fa6d7caf86a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42323 } } }
I20260924 19:05:06.167856 21060 sys_catalog.cc:458] T 00000000000000000000000000000000 P bde94082738d47eb914b2fa6d7caf86a [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:06.167508 21059 sys_catalog.cc:458] T 00000000000000000000000000000000 P bde94082738d47eb914b2fa6d7caf86a [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:06.169939 21064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:05:06.175050 21064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:05:06.183962 21064 catalog_manager.cc:1347] Generated new cluster ID: 12280b509ebe45f29fedb5a986675d71
I20260924 19:05:06.184070 21064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:05:06.197649 21064 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:05:06.198041 21064 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:05:06.209686 21064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bde94082738d47eb914b2fa6d7caf86a: Generated new TSK 0
I20260924 19:05:06.210467 21064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:05:06.227526 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 20981
W20260924 19:05:06.228585 21043 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40563
W20260924 19:05:06.229259 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 20981: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:05:06.229405 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 20981
2026-09-24T19:05:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1841 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:05:06 dist-test-slave-9fn1 krb5kdc[21079](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:05:06 dist-test-slave-9fn1 krb5kdc[21079](info): set up 2 sockets
Sep 24 19:05:06 dist-test-slave-9fn1 krb5kdc[21079](info): commencing operation
krb5kdc: starting...
W20260924 19:05:07.325800 388 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:05:07 dist-test-slave-9fn1 krb5kdc[21079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:05:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:05:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:05:07.565241 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36625
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:07.887490 21092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:07.887768 21092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:07.887859 21092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:07.894917 21092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:05:07.895021 21092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:07.895068 21092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:07.895128 21092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:05:07.895169 21092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:05:07.904742 21092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:07.906030 21092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:07.907254 21092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:07.919108 21097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:07.919195 21098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:07.919320 21100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:07.920109 21092 server_base.cc:1034] running on GCE node
I20260924 19:05:07.920845 21092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:07.922290 21092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:07.923519 21092 hybrid_clock.cc:648] HybridClock initialized: now 1790276707923482 us; error 56 us; skew 500 ppm
I20260924 19:05:07.923885 21092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:07 dist-test-slave-9fn1 krb5kdc[21079](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276707, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:07.926970 21092 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:05:07.928319 21092 webserver.cc:460] Webserver started at http://127.0.0.1:33271/ using document root <none> and password file <none>
I20260924 19:05:07.928936 21092 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:07.929036 21092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:07.929348 21092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:07.932197 21092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "8802c31587fe4f6a86016bc919451f2d"
format_stamp: "Formatted at 2026-09-24 19:05:07 on dist-test-slave-9fn1"
server_key: "033e09c15f01912b8827e435fb52af70"
server_key_iv: "bcaf8e5b6b1c8373a7c7f4033be2ce63"
server_key_version: "encryptionkey@0"
I20260924 19:05:07.932897 21092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "8802c31587fe4f6a86016bc919451f2d"
format_stamp: "Formatted at 2026-09-24 19:05:07 on dist-test-slave-9fn1"
server_key: "033e09c15f01912b8827e435fb52af70"
server_key_iv: "bcaf8e5b6b1c8373a7c7f4033be2ce63"
server_key_version: "encryptionkey@0"
I20260924 19:05:07.938061 21092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 19:05:07.941280 21107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:07.942561 21092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:05:07.942744 21092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "8802c31587fe4f6a86016bc919451f2d"
format_stamp: "Formatted at 2026-09-24 19:05:07 on dist-test-slave-9fn1"
server_key: "033e09c15f01912b8827e435fb52af70"
server_key_iv: "bcaf8e5b6b1c8373a7c7f4033be2ce63"
server_key_version: "encryptionkey@0"
I20260924 19:05:07.942935 21092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:07.966483 21092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:07.970958 21092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:07.971235 21092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:07.989100 21092 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36625
I20260924 19:05:07.989141 21158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36625 every 8 connection(s)
I20260924 19:05:07.990443 21092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:05:07.994050 21159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:05:08.000627 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 21092
I20260924 19:05:08.000823 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:05:08.001238 388 external_mini_cluster.cc:1428] Setting key 291423eb752bbb01a20dce1fd178855a
I20260924 19:05:08.004390 21159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8802c31587fe4f6a86016bc919451f2d: Bootstrap starting.
I20260924 19:05:08.007788 21159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8802c31587fe4f6a86016bc919451f2d: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:08.008934 21159 log.cc:824] T 00000000000000000000000000000000 P 8802c31587fe4f6a86016bc919451f2d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:05:08.011777 21159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8802c31587fe4f6a86016bc919451f2d: No bootstrap required, opened a new log
Sep 24 19:05:08 dist-test-slave-9fn1 krb5kdc[21079](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276707, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 19:05:08.020988 21159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8802c31587fe4f6a86016bc919451f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8802c31587fe4f6a86016bc919451f2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36625 } }
I20260924 19:05:08.021541 21159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8802c31587fe4f6a86016bc919451f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:05:08.021653 21159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8802c31587fe4f6a86016bc919451f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8802c31587fe4f6a86016bc919451f2d, State: Initialized, Role: FOLLOWER
I20260924 19:05:08.022310 21159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8802c31587fe4f6a86016bc919451f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8802c31587fe4f6a86016bc919451f2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36625 } }
I20260924 19:05:08.022567 21159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8802c31587fe4f6a86016bc919451f2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:05:08.022711 21159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8802c31587fe4f6a86016bc919451f2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:05:08.022917 21159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8802c31587fe4f6a86016bc919451f2d [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:08.026376 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:08.003137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55535 (local address 127.0.0.1:36625)
0924 19:05:08.003887 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:08.003898 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:08.003937 (+ 39us) server_negotiation.cc:407] Connection header received
0924 19:05:08.005702 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:08.005724 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:08.006095 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:08.006405 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:08.008119 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:08.009479 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:08.010913 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:08.011582 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:08.011778 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:08.016459 (+ 4681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:08.016467 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:08.016481 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:08.016525 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:08.020621 (+ 4096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:08.022378 (+ 1757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:08.022386 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:08.022395 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:08.022674 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:08.023092 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:08.023098 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:08.023101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:08.023818 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:08.024145 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:08.024492 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 19:05:08.024959 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":359,"threads_started":1}
I20260924 19:05:08.026261 21159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8802c31587fe4f6a86016bc919451f2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8802c31587fe4f6a86016bc919451f2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36625 } }
I20260924 19:05:08.026904 21159 leader_election.cc:302] T 00000000000000000000000000000000 P 8802c31587fe4f6a86016bc919451f2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8802c31587fe4f6a86016bc919451f2d; no voters:
I20260924 19:05:08.027447 21159 leader_election.cc:288] T 00000000000000000000000000000000 P 8802c31587fe4f6a86016bc919451f2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:05:08.027657 21164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8802c31587fe4f6a86016bc919451f2d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:05:08.029438 21164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8802c31587fe4f6a86016bc919451f2d [term 1 LEADER]: Becoming Leader. State: Replica: 8802c31587fe4f6a86016bc919451f2d, State: Running, Role: LEADER
I20260924 19:05:08.030021 21159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8802c31587fe4f6a86016bc919451f2d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:05:08.030561 21164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8802c31587fe4f6a86016bc919451f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8802c31587fe4f6a86016bc919451f2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36625 } }
I20260924 19:05:08.030812 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:05:08.033469 21165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8802c31587fe4f6a86016bc919451f2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8802c31587fe4f6a86016bc919451f2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8802c31587fe4f6a86016bc919451f2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36625 } } }
I20260924 19:05:08.034163 21165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8802c31587fe4f6a86016bc919451f2d [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:08.033607 21166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8802c31587fe4f6a86016bc919451f2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8802c31587fe4f6a86016bc919451f2d. Latest consensus state: current_term: 1 leader_uuid: "8802c31587fe4f6a86016bc919451f2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8802c31587fe4f6a86016bc919451f2d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36625 } } }
I20260924 19:05:08.034696 21166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8802c31587fe4f6a86016bc919451f2d [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:08.035740 21173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:05:08.047734 21173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:05:08.057931 21173 catalog_manager.cc:1347] Generated new cluster ID: f1082b5e019f4182932c915c11526aaf
I20260924 19:05:08.058038 21173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:05:08.072232 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:08.055480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41975 (local address 127.0.0.1:36625)
0924 19:05:08.055631 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:08.055639 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:08.056592 (+ 953us) server_negotiation.cc:407] Connection header received
0924 19:05:08.056871 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:08.056881 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:08.056960 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:08.057030 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:08.059043 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:08.060164 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:08.061960 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:08.062621 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:08.062728 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:08.065813 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:08.065819 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:08.065824 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:08.065859 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:08.069371 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:08.070183 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:08.070188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:08.070192 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:08.070352 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:08.070770 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:08.070774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:08.070777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:08.071110 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:08.071330 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:08.071707 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 19:05:08.071897 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 19:05:08.077816 21173 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:05:08.079103 21173 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:05:08.085300 21173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8802c31587fe4f6a86016bc919451f2d: Generated new TSK 0
I20260924 19:05:08.086284 21173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:05:08.092670 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 21092
I20260924 19:05:08.093669 21154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:05:08.183089 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 21092
2026-09-24T19:05:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1953 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:05:08 dist-test-slave-9fn1 krb5kdc[21191](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:05:08 dist-test-slave-9fn1 krb5kdc[21191](info): set up 2 sockets
Sep 24 19:05:08 dist-test-slave-9fn1 krb5kdc[21191](info): commencing operation
krb5kdc: starting...
W20260924 19:05:09.289858 388 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:05:09 dist-test-slave-9fn1 krb5kdc[21191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276709, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:05:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:05:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:05:09.528719 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33975
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:09.849649 21204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:09.849933 21204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:09.850121 21204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:09.857277 21204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:05:09.857385 21204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:09.857523 21204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:09.857632 21204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:05:09.857717 21204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:05:09.867449 21204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:09.868722 21204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:09.870029 21204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:09.881553 21209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:09.881492 21212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:09.881608 21210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:09.887166 21204 server_base.cc:1034] running on GCE node
I20260924 19:05:09.887964 21204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:09.889312 21204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:09.890508 21204 hybrid_clock.cc:648] HybridClock initialized: now 1790276709890455 us; error 37 us; skew 500 ppm
I20260924 19:05:09.890851 21204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:09 dist-test-slave-9fn1 krb5kdc[21191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:09.893899 21204 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:05:09.895397 21204 webserver.cc:460] Webserver started at http://127.0.0.1:43495/ using document root <none> and password file <none>
I20260924 19:05:09.896117 21204 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:09.896245 21204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:09.896554 21204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:09.899536 21204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "9ea705b276914c2e9ce5cbd0df7178bd"
format_stamp: "Formatted at 2026-09-24 19:05:09 on dist-test-slave-9fn1"
server_key: "622d1b97caf2296f3813dcede57e5e7f"
server_key_iv: "52a455ea5f483db2ede99e9fb9097719"
server_key_version: "encryptionkey@0"
I20260924 19:05:09.900308 21204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "9ea705b276914c2e9ce5cbd0df7178bd"
format_stamp: "Formatted at 2026-09-24 19:05:09 on dist-test-slave-9fn1"
server_key: "622d1b97caf2296f3813dcede57e5e7f"
server_key_iv: "52a455ea5f483db2ede99e9fb9097719"
server_key_version: "encryptionkey@0"
I20260924 19:05:09.905369 21204 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 19:05:09.908689 21219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:09.910027 21204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:05:09.910199 21204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "9ea705b276914c2e9ce5cbd0df7178bd"
format_stamp: "Formatted at 2026-09-24 19:05:09 on dist-test-slave-9fn1"
server_key: "622d1b97caf2296f3813dcede57e5e7f"
server_key_iv: "52a455ea5f483db2ede99e9fb9097719"
server_key_version: "encryptionkey@0"
I20260924 19:05:09.910392 21204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:09.925204 21204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:09.928982 21204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:09.929270 21204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:09.947213 21204 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33975
I20260924 19:05:09.947258 21270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33975 every 8 connection(s)
I20260924 19:05:09.948588 21204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:05:09.952241 21271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:05:09.956844 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 21204
I20260924 19:05:09.957051 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:05:09.957641 388 external_mini_cluster.cc:1428] Setting key 480731bde0d803451239f6c7cf547455
I20260924 19:05:09.963500 21271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ea705b276914c2e9ce5cbd0df7178bd: Bootstrap starting.
I20260924 19:05:09.967057 21271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ea705b276914c2e9ce5cbd0df7178bd: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:09.968338 21271 log.cc:824] T 00000000000000000000000000000000 P 9ea705b276914c2e9ce5cbd0df7178bd: Log is configured to *not* fsync() on all Append() calls
I20260924 19:05:09.971119 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:09.959355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46439 (local address 127.0.0.1:33975)
0924 19:05:09.960138 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:09.960149 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:09.960188 (+ 39us) server_negotiation.cc:407] Connection header received
0924 19:05:09.962051 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:09.962074 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:09.962446 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:09.962765 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:09.963309 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:09.964677 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:09.966579 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:09.967282 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:09.967410 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:09.968621 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:09.968628 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:09.968642 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:05:09.969409 (+ 767us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:09.969552 (+ 143us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:05:09.969588 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:09.969717 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:09.969924 (+ 207us) server_negotiation.cc:299] Negotiation successful
0924 19:05:09.970203 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":368,"threads_started":1}
I20260924 19:05:09.971375 21271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ea705b276914c2e9ce5cbd0df7178bd: No bootstrap required, opened a new log
I20260924 19:05:09.974618 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:05:09.979374 21271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ea705b276914c2e9ce5cbd0df7178bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea705b276914c2e9ce5cbd0df7178bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33975 } }
I20260924 19:05:09.979835 21271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ea705b276914c2e9ce5cbd0df7178bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:05:09.979948 21271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ea705b276914c2e9ce5cbd0df7178bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ea705b276914c2e9ce5cbd0df7178bd, State: Initialized, Role: FOLLOWER
I20260924 19:05:09.980566 21271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ea705b276914c2e9ce5cbd0df7178bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea705b276914c2e9ce5cbd0df7178bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33975 } }
I20260924 19:05:09.980805 21271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ea705b276914c2e9ce5cbd0df7178bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:05:09.980969 21271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ea705b276914c2e9ce5cbd0df7178bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:05:09.981153 21271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ea705b276914c2e9ce5cbd0df7178bd [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:09.984189 21271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ea705b276914c2e9ce5cbd0df7178bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea705b276914c2e9ce5cbd0df7178bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33975 } }
I20260924 19:05:09.984774 21271 leader_election.cc:302] T 00000000000000000000000000000000 P 9ea705b276914c2e9ce5cbd0df7178bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ea705b276914c2e9ce5cbd0df7178bd; no voters:
I20260924 19:05:09.985455 21271 leader_election.cc:288] T 00000000000000000000000000000000 P 9ea705b276914c2e9ce5cbd0df7178bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:05:09.985597 21281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ea705b276914c2e9ce5cbd0df7178bd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:05:09.985822 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:09.978867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39541 (local address 127.0.0.1:33975)
0924 19:05:09.979079 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:09.979087 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:09.979370 (+ 283us) server_negotiation.cc:407] Connection header received
0924 19:05:09.979559 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:09.979563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:09.979618 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:09.979695 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:09.980361 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:09.981248 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:09.983180 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:09.983622 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:09.983714 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:09.984860 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:09.984865 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:09.984868 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:05:09.984883 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:09.984941 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:05:09.984965 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:09.985049 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:09.985206 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 19:05:09.985320 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 19:05:09.986378 21281 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ea705b276914c2e9ce5cbd0df7178bd [term 1 LEADER]: Becoming Leader. State: Replica: 9ea705b276914c2e9ce5cbd0df7178bd, State: Running, Role: LEADER
I20260924 19:05:09.987043 21281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ea705b276914c2e9ce5cbd0df7178bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea705b276914c2e9ce5cbd0df7178bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33975 } }
I20260924 19:05:09.988719 21281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ea705b276914c2e9ce5cbd0df7178bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ea705b276914c2e9ce5cbd0df7178bd. Latest consensus state: current_term: 1 leader_uuid: "9ea705b276914c2e9ce5cbd0df7178bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea705b276914c2e9ce5cbd0df7178bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33975 } } }
I20260924 19:05:09.989033 21281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ea705b276914c2e9ce5cbd0df7178bd [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:09.990080 21283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ea705b276914c2e9ce5cbd0df7178bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ea705b276914c2e9ce5cbd0df7178bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ea705b276914c2e9ce5cbd0df7178bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33975 } } }
I20260924 19:05:09.990445 21283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ea705b276914c2e9ce5cbd0df7178bd [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:09.992708 21271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ea705b276914c2e9ce5cbd0df7178bd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:05:09.992841 21285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:05:09.998283 21285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:05:10.007920 21285 catalog_manager.cc:1347] Generated new cluster ID: c774b33e37884890aa04c414183aae93
I20260924 19:05:10.008047 21285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:05:10.017565 21285 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:05:10.019448 21285 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:05:10.026592 21285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ea705b276914c2e9ce5cbd0df7178bd: Generated new TSK 0
I20260924 19:05:10.027596 21285 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:05:10.033963 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 21204
W20260924 19:05:10.034945 21266 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46439
W20260924 19:05:10.035620 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 21204: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:05:10.035769 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 21204
2026-09-24T19:05:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1852 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:05:10 dist-test-slave-9fn1 krb5kdc[21302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:05:10 dist-test-slave-9fn1 krb5kdc[21302](info): set up 2 sockets
Sep 24 19:05:10 dist-test-slave-9fn1 krb5kdc[21302](info): commencing operation
krb5kdc: starting...
W20260924 19:05:11.141875 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:05:11 dist-test-slave-9fn1 krb5kdc[21302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:05:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:05:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:05:11.377264 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41603
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:11.708704 21315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:11.709002 21315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:11.709121 21315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:11.716352 21315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:05:11.716468 21315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:11.716519 21315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:11.716612 21315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:05:11.716679 21315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:05:11.727118 21315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:11.728420 21315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:11.729777 21315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:11.741549 21321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:11.741554 21323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:11.742045 21320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:11.742914 21315 server_base.cc:1034] running on GCE node
I20260924 19:05:11.743662 21315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:11.745007 21315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:11.746228 21315 hybrid_clock.cc:648] HybridClock initialized: now 1790276711746188 us; error 51 us; skew 500 ppm
I20260924 19:05:11.746592 21315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:11 dist-test-slave-9fn1 krb5kdc[21302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:11.749590 21315 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:05:11.751025 21315 webserver.cc:460] Webserver started at http://127.0.0.1:35515/ using document root <none> and password file <none>
I20260924 19:05:11.751690 21315 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:11.751832 21315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:11.752135 21315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:11.755053 21315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "0daaf02d50d942b1afabeb6f680f098b"
format_stamp: "Formatted at 2026-09-24 19:05:11 on dist-test-slave-9fn1"
server_key: "7854140fe3a01d59aeb0770f5700acea"
server_key_iv: "5ec2d7fbcbc084f213957bf5c495c89d"
server_key_version: "encryptionkey@0"
I20260924 19:05:11.755816 21315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "0daaf02d50d942b1afabeb6f680f098b"
format_stamp: "Formatted at 2026-09-24 19:05:11 on dist-test-slave-9fn1"
server_key: "7854140fe3a01d59aeb0770f5700acea"
server_key_iv: "5ec2d7fbcbc084f213957bf5c495c89d"
server_key_version: "encryptionkey@0"
I20260924 19:05:11.760790 21315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:05:11.764176 21330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:11.765386 21315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:05:11.765633 21315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "0daaf02d50d942b1afabeb6f680f098b"
format_stamp: "Formatted at 2026-09-24 19:05:11 on dist-test-slave-9fn1"
server_key: "7854140fe3a01d59aeb0770f5700acea"
server_key_iv: "5ec2d7fbcbc084f213957bf5c495c89d"
server_key_version: "encryptionkey@0"
I20260924 19:05:11.765846 21315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:11.771416 21315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:11.775243 21315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:11.775554 21315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:11.793972 21315 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41603
I20260924 19:05:11.794028 21381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41603 every 8 connection(s)
I20260924 19:05:11.795399 21315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:05:11.799028 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 21315
I20260924 19:05:11.799233 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:05:11.799224 21382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:05:11.799762 388 external_mini_cluster.cc:1428] Setting key 527e3e25c98a3773849a5d257d2a86c0
I20260924 19:05:11.807374 21385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:11.801729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50031 (local address 127.0.0.1:41603)
0924 19:05:11.802613 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:11.802623 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:11.802668 (+ 45us) server_negotiation.cc:407] Connection header received
0924 19:05:11.804157 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:11.804177 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:11.804491 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:11.804818 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:11.805881 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:11.805887 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:11.805899 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:05:11.805912 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:11.805972 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:05:11.805990 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:11.806157 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:11.806265 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 19:05:11.806461 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":404,"threads_started":1}
I20260924 19:05:11.810832 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:05:11.812664 21382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0daaf02d50d942b1afabeb6f680f098b: Bootstrap starting.
I20260924 19:05:11.816617 21382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0daaf02d50d942b1afabeb6f680f098b: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:11.817974 21382 log.cc:824] T 00000000000000000000000000000000 P 0daaf02d50d942b1afabeb6f680f098b: Log is configured to *not* fsync() on all Append() calls
I20260924 19:05:11.818331 21385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:11.815905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44723 (local address 127.0.0.1:41603)
0924 19:05:11.816042 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:11.816046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:11.816391 (+ 345us) server_negotiation.cc:407] Connection header received
0924 19:05:11.816571 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:11.816574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:11.816621 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:11.816717 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:11.817605 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:11.817610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:11.817612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:05:11.817628 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:11.817662 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:05:11.817676 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:11.817760 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:11.817907 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 19:05:11.817950 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 19:05:11.821033 21382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0daaf02d50d942b1afabeb6f680f098b: No bootstrap required, opened a new log
I20260924 19:05:11.828238 21382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0daaf02d50d942b1afabeb6f680f098b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0daaf02d50d942b1afabeb6f680f098b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41603 } }
I20260924 19:05:11.828631 21382 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0daaf02d50d942b1afabeb6f680f098b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:05:11.828744 21382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0daaf02d50d942b1afabeb6f680f098b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0daaf02d50d942b1afabeb6f680f098b, State: Initialized, Role: FOLLOWER
I20260924 19:05:11.829272 21382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0daaf02d50d942b1afabeb6f680f098b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0daaf02d50d942b1afabeb6f680f098b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41603 } }
I20260924 19:05:11.829526 21382 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0daaf02d50d942b1afabeb6f680f098b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:05:11.829686 21382 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0daaf02d50d942b1afabeb6f680f098b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:05:11.829859 21382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0daaf02d50d942b1afabeb6f680f098b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:11.831919 21382 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0daaf02d50d942b1afabeb6f680f098b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0daaf02d50d942b1afabeb6f680f098b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41603 } }
I20260924 19:05:11.832386 21382 leader_election.cc:302] T 00000000000000000000000000000000 P 0daaf02d50d942b1afabeb6f680f098b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0daaf02d50d942b1afabeb6f680f098b; no voters:
I20260924 19:05:11.832934 21382 leader_election.cc:288] T 00000000000000000000000000000000 P 0daaf02d50d942b1afabeb6f680f098b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:05:11.833096 21392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0daaf02d50d942b1afabeb6f680f098b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:05:11.833786 21392 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0daaf02d50d942b1afabeb6f680f098b [term 1 LEADER]: Becoming Leader. State: Replica: 0daaf02d50d942b1afabeb6f680f098b, State: Running, Role: LEADER
I20260924 19:05:11.834612 21392 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0daaf02d50d942b1afabeb6f680f098b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0daaf02d50d942b1afabeb6f680f098b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41603 } }
I20260924 19:05:11.835093 21382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0daaf02d50d942b1afabeb6f680f098b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:05:11.837648 21393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0daaf02d50d942b1afabeb6f680f098b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0daaf02d50d942b1afabeb6f680f098b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0daaf02d50d942b1afabeb6f680f098b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41603 } } }
I20260924 19:05:11.837625 21394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0daaf02d50d942b1afabeb6f680f098b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0daaf02d50d942b1afabeb6f680f098b. Latest consensus state: current_term: 1 leader_uuid: "0daaf02d50d942b1afabeb6f680f098b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0daaf02d50d942b1afabeb6f680f098b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41603 } } }
I20260924 19:05:11.838068 21393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0daaf02d50d942b1afabeb6f680f098b [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:11.838155 21394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0daaf02d50d942b1afabeb6f680f098b [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:11.840238 21400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:05:11.844712 21400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:05:11.853965 21400 catalog_manager.cc:1347] Generated new cluster ID: dc9df0ada0814108be9118d1e80738da
I20260924 19:05:11.854077 21400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:05:11.874095 21400 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:05:11.874506 21400 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:05:11.883862 21400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0daaf02d50d942b1afabeb6f680f098b: Generated new TSK 0
I20260924 19:05:11.884804 21400 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:05:11.896690 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 21315
W20260924 19:05:11.897797 21377 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50031
W20260924 19:05:11.898576 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 21315: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:05:11.898696 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 21315
2026-09-24T19:05:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1863 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:05:11 dist-test-slave-9fn1 krb5kdc[21413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:05:11 dist-test-slave-9fn1 krb5kdc[21413](info): set up 2 sockets
Sep 24 19:05:11 dist-test-slave-9fn1 krb5kdc[21413](info): commencing operation
krb5kdc: starting...
W20260924 19:05:12.989698 388 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:05:13 dist-test-slave-9fn1 krb5kdc[21413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:05:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:05:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:05:13.230861 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34327
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:13.550230 21426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:13.550570 21426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:13.550740 21426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:13.557935 21426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:05:13.558103 21426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:13.558214 21426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:13.558327 21426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:05:13.558414 21426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:05:13.568073 21426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:13.569355 21426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:13.570727 21426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:13.582149 21431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:13.582177 21432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:13.582234 21434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:13.582825 21426 server_base.cc:1034] running on GCE node
I20260924 19:05:13.583619 21426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:13.584970 21426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:13.586169 21426 hybrid_clock.cc:648] HybridClock initialized: now 1790276713586132 us; error 27 us; skew 500 ppm
I20260924 19:05:13.586506 21426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:13 dist-test-slave-9fn1 krb5kdc[21413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276713, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:13.589666 21426 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:05:13.591066 21426 webserver.cc:460] Webserver started at http://127.0.0.1:40711/ using document root <none> and password file <none>
I20260924 19:05:13.591753 21426 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:13.591884 21426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:13.592197 21426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:13.595140 21426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "df00717b6d9b4a34a53b9e36700c22c1"
format_stamp: "Formatted at 2026-09-24 19:05:13 on dist-test-slave-9fn1"
server_key: "758615b7fded7c4f06ccac16b25fb6f2"
server_key_iv: "3f3c2eb55ec1dc046bddd6df629fc4c7"
server_key_version: "encryptionkey@0"
I20260924 19:05:13.595920 21426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "df00717b6d9b4a34a53b9e36700c22c1"
format_stamp: "Formatted at 2026-09-24 19:05:13 on dist-test-slave-9fn1"
server_key: "758615b7fded7c4f06ccac16b25fb6f2"
server_key_iv: "3f3c2eb55ec1dc046bddd6df629fc4c7"
server_key_version: "encryptionkey@0"
I20260924 19:05:13.600788 21426 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:05:13.604128 21441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:13.605386 21426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:05:13.605588 21426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "df00717b6d9b4a34a53b9e36700c22c1"
format_stamp: "Formatted at 2026-09-24 19:05:13 on dist-test-slave-9fn1"
server_key: "758615b7fded7c4f06ccac16b25fb6f2"
server_key_iv: "3f3c2eb55ec1dc046bddd6df629fc4c7"
server_key_version: "encryptionkey@0"
I20260924 19:05:13.605753 21426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:13.628296 21426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:13.632478 21426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:13.632791 21426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:13.651453 21426 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34327
I20260924 19:05:13.651487 21492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34327 every 8 connection(s)
I20260924 19:05:13.652824 21426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:05:13.654464 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 21426
I20260924 19:05:13.654668 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:05:13.655073 388 external_mini_cluster.cc:1428] Setting key 5fac3f9dd7c756652ce6863c98759cd8
I20260924 19:05:13.657058 21493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 19:05:13 dist-test-slave-9fn1 krb5kdc[21413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 19:05:13.670482 21493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df00717b6d9b4a34a53b9e36700c22c1: Bootstrap starting.
I20260924 19:05:13.674683 21493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df00717b6d9b4a34a53b9e36700c22c1: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:13.675860 21493 log.cc:824] T 00000000000000000000000000000000 P df00717b6d9b4a34a53b9e36700c22c1: Log is configured to *not* fsync() on all Append() calls
I20260924 19:05:13.679090 21493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df00717b6d9b4a34a53b9e36700c22c1: No bootstrap required, opened a new log
I20260924 19:05:13.679126 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:13.656799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.207:44545 (local address 127.0.0.1:34327)
0924 19:05:13.657640 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:13.657652 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:13.657692 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:05:13.659323 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:13.659342 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:13.659728 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:13.660051 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:13.661849 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:13.663195 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:13.665092 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:13.665812 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:13.665940 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:13.670781 (+ 4841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:13.670801 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:13.670822 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:13.670871 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:13.674685 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:13.675454 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:13.675462 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:13.675471 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:13.675697 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:13.676123 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:13.676127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:13.676130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:13.676768 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:13.676993 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:13.677597 (+ 604us) server_negotiation.cc:299] Negotiation successful
0924 19:05:13.677897 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":386,"threads_started":1}
I20260924 19:05:13.682852 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:05:13.687268 21493 raft_consensus.cc:348] T 00000000000000000000000000000000 P df00717b6d9b4a34a53b9e36700c22c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df00717b6d9b4a34a53b9e36700c22c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34327 } }
I20260924 19:05:13.687613 21493 raft_consensus.cc:374] T 00000000000000000000000000000000 P df00717b6d9b4a34a53b9e36700c22c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:05:13.687737 21493 raft_consensus.cc:729] T 00000000000000000000000000000000 P df00717b6d9b4a34a53b9e36700c22c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df00717b6d9b4a34a53b9e36700c22c1, State: Initialized, Role: FOLLOWER
I20260924 19:05:13.688321 21493 consensus_queue.cc:260] T 00000000000000000000000000000000 P df00717b6d9b4a34a53b9e36700c22c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df00717b6d9b4a34a53b9e36700c22c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34327 } }
I20260924 19:05:13.688565 21493 raft_consensus.cc:388] T 00000000000000000000000000000000 P df00717b6d9b4a34a53b9e36700c22c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:05:13.688701 21493 raft_consensus.cc:482] T 00000000000000000000000000000000 P df00717b6d9b4a34a53b9e36700c22c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:05:13.688875 21493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df00717b6d9b4a34a53b9e36700c22c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:13.691135 21493 raft_consensus.cc:504] T 00000000000000000000000000000000 P df00717b6d9b4a34a53b9e36700c22c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df00717b6d9b4a34a53b9e36700c22c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34327 } }
I20260924 19:05:13.691609 21493 leader_election.cc:302] T 00000000000000000000000000000000 P df00717b6d9b4a34a53b9e36700c22c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df00717b6d9b4a34a53b9e36700c22c1; no voters:
I20260924 19:05:13.692278 21493 leader_election.cc:288] T 00000000000000000000000000000000 P df00717b6d9b4a34a53b9e36700c22c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:05:13.692376 21503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df00717b6d9b4a34a53b9e36700c22c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:05:13.694725 21493 sys_catalog.cc:564] T 00000000000000000000000000000000 P df00717b6d9b4a34a53b9e36700c22c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:05:13.699389 21503 raft_consensus.cc:686] T 00000000000000000000000000000000 P df00717b6d9b4a34a53b9e36700c22c1 [term 1 LEADER]: Becoming Leader. State: Replica: df00717b6d9b4a34a53b9e36700c22c1, State: Running, Role: LEADER
I20260924 19:05:13.700152 21503 consensus_queue.cc:237] T 00000000000000000000000000000000 P df00717b6d9b4a34a53b9e36700c22c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df00717b6d9b4a34a53b9e36700c22c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34327 } }
I20260924 19:05:13.701347 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:13.686977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.207:43669 (local address 127.0.0.1:34327)
0924 19:05:13.687155 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:13.687161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:13.687656 (+ 495us) server_negotiation.cc:407] Connection header received
0924 19:05:13.687831 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:13.687838 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:13.687912 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:13.688010 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:13.689558 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:13.690373 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:13.691668 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:13.692163 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:13.692335 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:13.695188 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:13.695193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:13.695196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:13.695225 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:13.698455 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:13.699182 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:13.699187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:13.699195 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:13.699362 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:13.699867 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:13.699871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:13.699873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:13.700161 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:13.700489 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:13.700806 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 19:05:13.700984 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 19:05:13.703709 21504 sys_catalog.cc:455] T 00000000000000000000000000000000 P df00717b6d9b4a34a53b9e36700c22c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df00717b6d9b4a34a53b9e36700c22c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df00717b6d9b4a34a53b9e36700c22c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34327 } } }
I20260924 19:05:13.704349 21504 sys_catalog.cc:458] T 00000000000000000000000000000000 P df00717b6d9b4a34a53b9e36700c22c1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:13.705601 21505 sys_catalog.cc:455] T 00000000000000000000000000000000 P df00717b6d9b4a34a53b9e36700c22c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df00717b6d9b4a34a53b9e36700c22c1. Latest consensus state: current_term: 1 leader_uuid: "df00717b6d9b4a34a53b9e36700c22c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df00717b6d9b4a34a53b9e36700c22c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34327 } } }
I20260924 19:05:13.705873 21505 sys_catalog.cc:458] T 00000000000000000000000000000000 P df00717b6d9b4a34a53b9e36700c22c1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:13.707870 21512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:05:13.713665 21512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:05:13.724118 21512 catalog_manager.cc:1347] Generated new cluster ID: f42bb021b7b8441ea73765e4d0828a1b
I20260924 19:05:13.724254 21512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:05:13.763696 21512 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:05:13.765625 21512 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:05:13.772306 21512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df00717b6d9b4a34a53b9e36700c22c1: Generated new TSK 0
I20260924 19:05:13.773293 21512 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:05:13.780853 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 21426
I20260924 19:05:13.783093 21488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:05:13.874336 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 21426
W20260924 19:05:13.891618 21411 connection.cc:535] client connection to 127.0.0.1:34327 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34327): Connection reset by peer (error 104)
2026-09-24T19:05:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1975 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:05:13 dist-test-slave-9fn1 krb5kdc[21525](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:05:13 dist-test-slave-9fn1 krb5kdc[21525](info): set up 2 sockets
Sep 24 19:05:13 dist-test-slave-9fn1 krb5kdc[21525](info): commencing operation
krb5kdc: starting...
W20260924 19:05:14.965848 388 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:05:15 dist-test-slave-9fn1 krb5kdc[21525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:05:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:05:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:05:15.204803 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32833
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:15.524199 21538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:15.524528 21538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:15.524708 21538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:15.532015 21538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:05:15.532162 21538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:15.532271 21538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:15.532382 21538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:05:15.532464 21538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:05:15.542264 21538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:15.543598 21538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:15.544960 21538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:15.556999 21543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:15.556888 21546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:15.556973 21544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:15.557878 21538 server_base.cc:1034] running on GCE node
I20260924 19:05:15.558753 21538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:15.560160 21538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:15.561415 21538 hybrid_clock.cc:648] HybridClock initialized: now 1790276715561375 us; error 46 us; skew 500 ppm
I20260924 19:05:15.561848 21538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:15 dist-test-slave-9fn1 krb5kdc[21525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276715, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:15.564898 21538 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:05:15.566383 21538 webserver.cc:460] Webserver started at http://127.0.0.1:36721/ using document root <none> and password file <none>
I20260924 19:05:15.567046 21538 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:15.567189 21538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:15.567487 21538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:15.570480 21538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "3e3bc3acf9504822b534763d68cd83f8"
format_stamp: "Formatted at 2026-09-24 19:05:15 on dist-test-slave-9fn1"
server_key: "dfa15c35ceb535e09e146b0b49754a39"
server_key_iv: "fb47d09902358b281eac7e108b63cbee"
server_key_version: "encryptionkey@0"
I20260924 19:05:15.571219 21538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "3e3bc3acf9504822b534763d68cd83f8"
format_stamp: "Formatted at 2026-09-24 19:05:15 on dist-test-slave-9fn1"
server_key: "dfa15c35ceb535e09e146b0b49754a39"
server_key_iv: "fb47d09902358b281eac7e108b63cbee"
server_key_version: "encryptionkey@0"
I20260924 19:05:15.576216 21538 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:05:15.579640 21553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:15.580893 21538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:05:15.581069 21538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "3e3bc3acf9504822b534763d68cd83f8"
format_stamp: "Formatted at 2026-09-24 19:05:15 on dist-test-slave-9fn1"
server_key: "dfa15c35ceb535e09e146b0b49754a39"
server_key_iv: "fb47d09902358b281eac7e108b63cbee"
server_key_version: "encryptionkey@0"
I20260924 19:05:15.581271 21538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:15.594383 21538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:15.598027 21538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:15.598304 21538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:15.616029 21538 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32833
I20260924 19:05:15.616066 21604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32833 every 8 connection(s)
I20260924 19:05:15.617460 21538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:05:15.618080 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 21538
I20260924 19:05:15.618295 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:05:15.618857 388 external_mini_cluster.cc:1428] Setting key f58b761fe49f1fcab43e4121635f6013
I20260924 19:05:15.621820 21605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:05:15.631641 21608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:15.620600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.207:46465 (local address 127.0.0.1:32833)
0924 19:05:15.621322 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:15.621341 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:15.621385 (+ 44us) server_negotiation.cc:407] Connection header received
0924 19:05:15.623148 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:15.623170 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:15.623556 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:15.623920 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:15.624513 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:15.625867 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:15.627192 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:15.627857 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:15.628058 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:15.629270 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:15.629280 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:15.629293 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:05:15.629938 (+ 645us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:15.630044 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:05:15.630078 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:15.630262 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:15.630569 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 19:05:15.630725 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":299,"threads_started":1}
I20260924 19:05:15.632601 21605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e3bc3acf9504822b534763d68cd83f8: Bootstrap starting.
I20260924 19:05:15.635433 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:05:15.635953 21605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e3bc3acf9504822b534763d68cd83f8: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:15.636922 21605 log.cc:824] T 00000000000000000000000000000000 P 3e3bc3acf9504822b534763d68cd83f8: Log is configured to *not* fsync() on all Append() calls
I20260924 19:05:15.639654 21605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e3bc3acf9504822b534763d68cd83f8: No bootstrap required, opened a new log
I20260924 19:05:15.646921 21608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:15.639850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.207:57809 (local address 127.0.0.1:32833)
0924 19:05:15.640083 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:15.640092 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:15.640546 (+ 454us) server_negotiation.cc:407] Connection header received
0924 19:05:15.640718 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:15.640722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:15.640779 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:15.640868 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:15.641596 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:15.642402 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:15.644286 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:15.644712 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:15.644805 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:15.646114 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:15.646119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:15.646125 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:05:15.646146 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:15.646187 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:05:15.646202 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:15.646285 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:15.646498 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 19:05:15.646542 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 19:05:15.649366 21605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e3bc3acf9504822b534763d68cd83f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3bc3acf9504822b534763d68cd83f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32833 } }
I20260924 19:05:15.649839 21605 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e3bc3acf9504822b534763d68cd83f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:05:15.649950 21605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e3bc3acf9504822b534763d68cd83f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e3bc3acf9504822b534763d68cd83f8, State: Initialized, Role: FOLLOWER
I20260924 19:05:15.650583 21605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e3bc3acf9504822b534763d68cd83f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3bc3acf9504822b534763d68cd83f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32833 } }
I20260924 19:05:15.650826 21605 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e3bc3acf9504822b534763d68cd83f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:05:15.650985 21605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e3bc3acf9504822b534763d68cd83f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:05:15.651160 21605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e3bc3acf9504822b534763d68cd83f8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:15.653339 21605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e3bc3acf9504822b534763d68cd83f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3bc3acf9504822b534763d68cd83f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32833 } }
I20260924 19:05:15.653887 21605 leader_election.cc:302] T 00000000000000000000000000000000 P 3e3bc3acf9504822b534763d68cd83f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e3bc3acf9504822b534763d68cd83f8; no voters:
I20260924 19:05:15.654558 21605 leader_election.cc:288] T 00000000000000000000000000000000 P 3e3bc3acf9504822b534763d68cd83f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:05:15.654732 21615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e3bc3acf9504822b534763d68cd83f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:05:15.655450 21615 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e3bc3acf9504822b534763d68cd83f8 [term 1 LEADER]: Becoming Leader. State: Replica: 3e3bc3acf9504822b534763d68cd83f8, State: Running, Role: LEADER
I20260924 19:05:15.656034 21615 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e3bc3acf9504822b534763d68cd83f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3bc3acf9504822b534763d68cd83f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32833 } }
I20260924 19:05:15.656723 21605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e3bc3acf9504822b534763d68cd83f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:05:15.658813 21616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e3bc3acf9504822b534763d68cd83f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e3bc3acf9504822b534763d68cd83f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3bc3acf9504822b534763d68cd83f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32833 } } }
I20260924 19:05:15.659268 21616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e3bc3acf9504822b534763d68cd83f8 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:15.662158 21617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e3bc3acf9504822b534763d68cd83f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e3bc3acf9504822b534763d68cd83f8. Latest consensus state: current_term: 1 leader_uuid: "3e3bc3acf9504822b534763d68cd83f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3bc3acf9504822b534763d68cd83f8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32833 } } }
I20260924 19:05:15.662478 21617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e3bc3acf9504822b534763d68cd83f8 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:15.662599 21623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:05:15.667639 21623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:05:15.677158 21623 catalog_manager.cc:1347] Generated new cluster ID: d4fdfbaac3b04797861923b71634e761
I20260924 19:05:15.677265 21623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:05:15.684866 21623 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:05:15.686194 21623 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:05:15.708745 21623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e3bc3acf9504822b534763d68cd83f8: Generated new TSK 0
I20260924 19:05:15.709669 21623 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:05:15.725515 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 21538
W20260924 19:05:15.726737 21600 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.207:46465
W20260924 19:05:15.727452 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 21538: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:05:15.727589 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 21538
W20260924 19:05:15.744362 21523 connection.cc:535] client connection to 127.0.0.1:32833 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32833): Connection reset by peer (error 104)
2026-09-24T19:05:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1852 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:05:15 dist-test-slave-9fn1 krb5kdc[21636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:05:15 dist-test-slave-9fn1 krb5kdc[21636](info): set up 2 sockets
Sep 24 19:05:15 dist-test-slave-9fn1 krb5kdc[21636](info): commencing operation
krb5kdc: starting...
W20260924 19:05:16.837852 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:05:17 dist-test-slave-9fn1 krb5kdc[21636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:05:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:05:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:05:17.084827 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44555
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:17.401414 21649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:17.401782 21649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:17.401944 21649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:17.409296 21649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:05:17.409442 21649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:17.409565 21649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:17.409678 21649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:05:17.409766 21649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:05:17.419564 21649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:17.420747 21649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:17.422041 21649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:17.433563 21654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:17.433622 21657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:17.433588 21655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:17.434481 21649 server_base.cc:1034] running on GCE node
I20260924 19:05:17.435179 21649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:17.436532 21649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:17.437752 21649 hybrid_clock.cc:648] HybridClock initialized: now 1790276717437722 us; error 44 us; skew 500 ppm
I20260924 19:05:17.438093 21649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:17 dist-test-slave-9fn1 krb5kdc[21636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:17.446952 21649 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:05:17.448513 21649 webserver.cc:460] Webserver started at http://127.0.0.1:32811/ using document root <none> and password file <none>
I20260924 19:05:17.449190 21649 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:17.449340 21649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:17.449723 21649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:17.452661 21649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "1be03c5324a948d79b1504bf129558e2"
format_stamp: "Formatted at 2026-09-24 19:05:17 on dist-test-slave-9fn1"
server_key: "c52947693d73526624e87b09d984a0e3"
server_key_iv: "97df047ea647aab38bedcd1ede27ca1c"
server_key_version: "encryptionkey@0"
I20260924 19:05:17.453418 21649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "1be03c5324a948d79b1504bf129558e2"
format_stamp: "Formatted at 2026-09-24 19:05:17 on dist-test-slave-9fn1"
server_key: "c52947693d73526624e87b09d984a0e3"
server_key_iv: "97df047ea647aab38bedcd1ede27ca1c"
server_key_version: "encryptionkey@0"
I20260924 19:05:17.458580 21649 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 19:05:17.462018 21664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:17.463303 21649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 19:05:17.463483 21649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "1be03c5324a948d79b1504bf129558e2"
format_stamp: "Formatted at 2026-09-24 19:05:17 on dist-test-slave-9fn1"
server_key: "c52947693d73526624e87b09d984a0e3"
server_key_iv: "97df047ea647aab38bedcd1ede27ca1c"
server_key_version: "encryptionkey@0"
I20260924 19:05:17.463729 21649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:17.469229 21649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:17.473013 21649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:17.473307 21649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:17.491159 21649 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44555
I20260924 19:05:17.491202 21715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44555 every 8 connection(s)
I20260924 19:05:17.492432 21649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:05:17.495903 21716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:05:17.498315 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 21649
I20260924 19:05:17.498486 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:05:17.499002 388 external_mini_cluster.cc:1428] Setting key ef036d431759784c0ec25123f3ae8ac9
I20260924 19:05:17.506491 21719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:17.500791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.207:53515 (local address 127.0.0.1:44555)
0924 19:05:17.501607 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:17.501617 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:17.501659 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:05:17.503284 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:17.503305 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:17.503608 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:17.503964 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:17.504969 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:17.504975 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:17.504986 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:05:17.505002 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:17.505056 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:05:17.505076 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:17.505323 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:17.505458 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 19:05:17.505670 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":318,"threads_started":1}
I20260924 19:05:17.507426 21716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1be03c5324a948d79b1504bf129558e2: Bootstrap starting.
I20260924 19:05:17.510383 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:05:17.511351 21716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1be03c5324a948d79b1504bf129558e2: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:17.512518 21716 log.cc:824] T 00000000000000000000000000000000 P 1be03c5324a948d79b1504bf129558e2: Log is configured to *not* fsync() on all Append() calls
I20260924 19:05:17.515720 21716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1be03c5324a948d79b1504bf129558e2: No bootstrap required, opened a new log
I20260924 19:05:17.517670 21719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:17.515471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.207:59441 (local address 127.0.0.1:44555)
0924 19:05:17.515620 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:17.515625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:17.515934 (+ 309us) server_negotiation.cc:407] Connection header received
0924 19:05:17.516061 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:17.516066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:17.516140 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:17.516269 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:17.517049 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:17.517056 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:17.517060 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:05:17.517076 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:17.517117 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:05:17.517136 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:17.517256 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:17.517312 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 19:05:17.517363 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 19:05:17.523734 21716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1be03c5324a948d79b1504bf129558e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be03c5324a948d79b1504bf129558e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44555 } }
I20260924 19:05:17.524047 21716 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1be03c5324a948d79b1504bf129558e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:05:17.524116 21716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1be03c5324a948d79b1504bf129558e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1be03c5324a948d79b1504bf129558e2, State: Initialized, Role: FOLLOWER
I20260924 19:05:17.524571 21716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1be03c5324a948d79b1504bf129558e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be03c5324a948d79b1504bf129558e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44555 } }
I20260924 19:05:17.524809 21716 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1be03c5324a948d79b1504bf129558e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:05:17.524945 21716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1be03c5324a948d79b1504bf129558e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:05:17.525138 21716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1be03c5324a948d79b1504bf129558e2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:17.527257 21716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1be03c5324a948d79b1504bf129558e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be03c5324a948d79b1504bf129558e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44555 } }
I20260924 19:05:17.527732 21716 leader_election.cc:302] T 00000000000000000000000000000000 P 1be03c5324a948d79b1504bf129558e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1be03c5324a948d79b1504bf129558e2; no voters:
I20260924 19:05:17.528254 21716 leader_election.cc:288] T 00000000000000000000000000000000 P 1be03c5324a948d79b1504bf129558e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:05:17.528427 21726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1be03c5324a948d79b1504bf129558e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:05:17.529134 21726 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1be03c5324a948d79b1504bf129558e2 [term 1 LEADER]: Becoming Leader. State: Replica: 1be03c5324a948d79b1504bf129558e2, State: Running, Role: LEADER
I20260924 19:05:17.529663 21726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1be03c5324a948d79b1504bf129558e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be03c5324a948d79b1504bf129558e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44555 } }
I20260924 19:05:17.530464 21716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1be03c5324a948d79b1504bf129558e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:05:17.532261 21728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be03c5324a948d79b1504bf129558e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1be03c5324a948d79b1504bf129558e2. Latest consensus state: current_term: 1 leader_uuid: "1be03c5324a948d79b1504bf129558e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be03c5324a948d79b1504bf129558e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44555 } } }
I20260924 19:05:17.532603 21728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be03c5324a948d79b1504bf129558e2 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:17.533468 21732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:05:17.533299 21727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be03c5324a948d79b1504bf129558e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1be03c5324a948d79b1504bf129558e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be03c5324a948d79b1504bf129558e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44555 } } }
I20260924 19:05:17.533701 21727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be03c5324a948d79b1504bf129558e2 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:17.539068 21732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:05:17.547940 21732 catalog_manager.cc:1347] Generated new cluster ID: 0a7471b85d044c5ea001cf9c2c30121c
I20260924 19:05:17.548065 21732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:05:17.581936 21732 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:05:17.582425 21732 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:05:17.591773 21732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1be03c5324a948d79b1504bf129558e2: Generated new TSK 0
I20260924 19:05:17.592706 21732 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:05:17.604048 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 21649
W20260924 19:05:17.605135 21711 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.207:53515
W20260924 19:05:17.605811 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 21649: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:05:17.605957 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 21649
2026-09-24T19:05:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1878 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:05:17 dist-test-slave-9fn1 krb5kdc[21747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:05:17 dist-test-slave-9fn1 krb5kdc[21747](info): set up 2 sockets
Sep 24 19:05:17 dist-test-slave-9fn1 krb5kdc[21747](info): commencing operation
krb5kdc: starting...
W20260924 19:05:18.693825 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:05:18 dist-test-slave-9fn1 krb5kdc[21747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:05:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:05:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:05:18.934782 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39171
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:19.274392 21760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:19.274751 21760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:19.274912 21760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:19.282128 21760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:05:19.282279 21760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:19.282382 21760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:19.282502 21760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:05:19.282582 21760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:05:19.292248 21760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39171
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:19.293570 21760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:19.294842 21760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:19.306799 21768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:19.306898 21766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:19.306866 21765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:19.307775 21760 server_base.cc:1034] running on GCE node
I20260924 19:05:19.308643 21760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:19.310178 21760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:19.311415 21760 hybrid_clock.cc:648] HybridClock initialized: now 1790276719311360 us; error 43 us; skew 500 ppm
I20260924 19:05:19.311765 21760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:19 dist-test-slave-9fn1 krb5kdc[21747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276719, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:19.314972 21760 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:05:19.316329 21760 webserver.cc:460] Webserver started at http://127.0.0.1:43039/ using document root <none> and password file <none>
I20260924 19:05:19.316949 21760 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:19.317051 21760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:19.317293 21760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:19.320312 21760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "ea88500f764046c3ae2263198736dfa1"
format_stamp: "Formatted at 2026-09-24 19:05:19 on dist-test-slave-9fn1"
server_key: "398e661cd560f2e58c20246ed40a1473"
server_key_iv: "d030a1857f23ff4d979440b455f3f6be"
server_key_version: "encryptionkey@0"
I20260924 19:05:19.321002 21760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "ea88500f764046c3ae2263198736dfa1"
format_stamp: "Formatted at 2026-09-24 19:05:19 on dist-test-slave-9fn1"
server_key: "398e661cd560f2e58c20246ed40a1473"
server_key_iv: "d030a1857f23ff4d979440b455f3f6be"
server_key_version: "encryptionkey@0"
I20260924 19:05:19.326192 21760 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260924 19:05:19.329437 21775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:19.330695 21760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 19:05:19.330832 21760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "ea88500f764046c3ae2263198736dfa1"
format_stamp: "Formatted at 2026-09-24 19:05:19 on dist-test-slave-9fn1"
server_key: "398e661cd560f2e58c20246ed40a1473"
server_key_iv: "d030a1857f23ff4d979440b455f3f6be"
server_key_version: "encryptionkey@0"
I20260924 19:05:19.330998 21760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:19.357270 21760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:19.361729 21760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:19.361970 21760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:19.379830 21760 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39171
I20260924 19:05:19.379873 21826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39171 every 8 connection(s)
I20260924 19:05:19.381192 21760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:05:19.385003 21827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:05:19.389158 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 21760
I20260924 19:05:19.389343 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:05:19.389811 388 external_mini_cluster.cc:1428] Setting key 13a44c36ff4ad8cfa60a0e44fe203e59
I20260924 19:05:19.396924 21827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea88500f764046c3ae2263198736dfa1: Bootstrap starting.
I20260924 19:05:19.399938 21827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea88500f764046c3ae2263198736dfa1: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:19.400878 21827 log.cc:824] T 00000000000000000000000000000000 P ea88500f764046c3ae2263198736dfa1: Log is configured to *not* fsync() on all Append() calls
I20260924 19:05:19.403457 21827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea88500f764046c3ae2263198736dfa1: No bootstrap required, opened a new log
Sep 24 19:05:19 dist-test-slave-9fn1 krb5kdc[21747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 19:05:19.411995 21827 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea88500f764046c3ae2263198736dfa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea88500f764046c3ae2263198736dfa1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39171 } }
I20260924 19:05:19.412533 21827 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea88500f764046c3ae2263198736dfa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:05:19.412671 21827 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea88500f764046c3ae2263198736dfa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea88500f764046c3ae2263198736dfa1, State: Initialized, Role: FOLLOWER
I20260924 19:05:19.413307 21827 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea88500f764046c3ae2263198736dfa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea88500f764046c3ae2263198736dfa1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39171 } }
I20260924 19:05:19.413578 21827 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea88500f764046c3ae2263198736dfa1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:05:19.413658 21830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:19.391414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.207:52311 (local address 127.0.0.1:39171)
0924 19:05:19.394169 (+ 2755us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:19.394189 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:19.394245 (+ 56us) server_negotiation.cc:407] Connection header received
0924 19:05:19.396125 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:19.396147 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:19.396520 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:19.396895 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:19.398401 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:19.399681 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:19.401002 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:19.401760 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:19.401964 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:19.406225 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:19.406232 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:19.406246 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:19.406282 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:19.409358 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:19.410288 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:19.410294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:19.410300 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:19.410470 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:19.410925 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:19.410929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:19.410930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:19.411420 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:19.411643 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:19.412196 (+ 553us) server_negotiation.cc:299] Negotiation successful
0924 19:05:19.412502 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2503,"thread_start_us":2303,"threads_started":1}
I20260924 19:05:19.413736 21827 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea88500f764046c3ae2263198736dfa1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:05:19.413946 21827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea88500f764046c3ae2263198736dfa1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:19.416236 21827 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea88500f764046c3ae2263198736dfa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea88500f764046c3ae2263198736dfa1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39171 } }
I20260924 19:05:19.416726 21827 leader_election.cc:302] T 00000000000000000000000000000000 P ea88500f764046c3ae2263198736dfa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea88500f764046c3ae2263198736dfa1; no voters:
I20260924 19:05:19.417376 21827 leader_election.cc:288] T 00000000000000000000000000000000 P ea88500f764046c3ae2263198736dfa1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:05:19.417524 21832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea88500f764046c3ae2263198736dfa1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:05:19.417609 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:05:19.419281 21827 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea88500f764046c3ae2263198736dfa1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:05:19.421066 21832 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea88500f764046c3ae2263198736dfa1 [term 1 LEADER]: Becoming Leader. State: Replica: ea88500f764046c3ae2263198736dfa1, State: Running, Role: LEADER
I20260924 19:05:19.421751 21832 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea88500f764046c3ae2263198736dfa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea88500f764046c3ae2263198736dfa1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39171 } }
I20260924 19:05:19.424940 21834 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea88500f764046c3ae2263198736dfa1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea88500f764046c3ae2263198736dfa1. Latest consensus state: current_term: 1 leader_uuid: "ea88500f764046c3ae2263198736dfa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea88500f764046c3ae2263198736dfa1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39171 } } }
I20260924 19:05:19.425673 21833 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea88500f764046c3ae2263198736dfa1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea88500f764046c3ae2263198736dfa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea88500f764046c3ae2263198736dfa1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39171 } } }
I20260924 19:05:19.426085 21833 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea88500f764046c3ae2263198736dfa1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:19.426702 21834 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea88500f764046c3ae2263198736dfa1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:19.434736 21842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:05:19.444330 21842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:05:19.456120 21830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:19.435679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.207:50825 (local address 127.0.0.1:39171)
0924 19:05:19.436001 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:19.436010 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:19.436730 (+ 720us) server_negotiation.cc:407] Connection header received
0924 19:05:19.436986 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:19.436995 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:19.437109 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:19.437260 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:19.439832 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:19.441169 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:19.443144 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:19.443775 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:19.443913 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:19.447401 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:19.447407 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:19.447412 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:19.447459 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:19.451139 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:19.452124 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:19.452129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:19.452132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:19.452299 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:19.453019 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:19.453027 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:19.453033 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:19.453601 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:19.453924 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:19.455317 (+ 1393us) server_negotiation.cc:299] Negotiation successful
0924 19:05:19.455568 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260924 19:05:19.464835 21842 catalog_manager.cc:1347] Generated new cluster ID: 3e95ce6f16f243ee85eb95ec085d5b9c
I20260924 19:05:19.464962 21842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:05:19.492926 21842 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:05:19.494300 21842 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:05:19.505939 21842 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea88500f764046c3ae2263198736dfa1: Generated new TSK 0
I20260924 19:05:19.506791 21842 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:05:19.515158 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 21760
I20260924 19:05:19.516223 21822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:05:19.610061 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 21760
W20260924 19:05:19.627489 21745 connection.cc:535] client connection to 127.0.0.1:39171 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39171): Connection reset by peer (error 104)
2026-09-24T19:05:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2004 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:05:19 dist-test-slave-9fn1 krb5kdc[21860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:05:19 dist-test-slave-9fn1 krb5kdc[21860](info): set up 2 sockets
Sep 24 19:05:19 dist-test-slave-9fn1 krb5kdc[21860](info): commencing operation
krb5kdc: starting...
W20260924 19:05:20.733771 388 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:05:20 dist-test-slave-9fn1 krb5kdc[21860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:05:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:05:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:05:20.976565 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40053
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:21.298796 21873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:21.299114 21873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:21.299273 21873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:21.306376 21873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:05:21.306515 21873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:21.306615 21873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:21.306720 21873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:05:21.306804 21873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:05:21.316157 21873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40053
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:21.317353 21873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:21.318687 21873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:21.330945 21878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:21.331045 21881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:21.331048 21879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:21.331833 21873 server_base.cc:1034] running on GCE node
I20260924 19:05:21.332707 21873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:21.334147 21873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:21.335398 21873 hybrid_clock.cc:648] HybridClock initialized: now 1790276721335375 us; error 55 us; skew 500 ppm
I20260924 19:05:21.335767 21873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:21 dist-test-slave-9fn1 krb5kdc[21860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:21.338980 21873 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:05:21.340569 21873 webserver.cc:460] Webserver started at http://127.0.0.1:42463/ using document root <none> and password file <none>
I20260924 19:05:21.341192 21873 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:21.341291 21873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:21.341607 21873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:21.344576 21873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "48eeccb82f3e4764a4bf6f2bd18d9a47"
format_stamp: "Formatted at 2026-09-24 19:05:21 on dist-test-slave-9fn1"
server_key: "29ad45d88341fccd25301b0ada3c2f9c"
server_key_iv: "a37bab32ac49192d7a2dbbbdbd37d96f"
server_key_version: "encryptionkey@0"
I20260924 19:05:21.345261 21873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "48eeccb82f3e4764a4bf6f2bd18d9a47"
format_stamp: "Formatted at 2026-09-24 19:05:21 on dist-test-slave-9fn1"
server_key: "29ad45d88341fccd25301b0ada3c2f9c"
server_key_iv: "a37bab32ac49192d7a2dbbbdbd37d96f"
server_key_version: "encryptionkey@0"
I20260924 19:05:21.350234 21873 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260924 19:05:21.353451 21888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:21.354789 21873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 19:05:21.354919 21873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "48eeccb82f3e4764a4bf6f2bd18d9a47"
format_stamp: "Formatted at 2026-09-24 19:05:21 on dist-test-slave-9fn1"
server_key: "29ad45d88341fccd25301b0ada3c2f9c"
server_key_iv: "a37bab32ac49192d7a2dbbbdbd37d96f"
server_key_version: "encryptionkey@0"
I20260924 19:05:21.355072 21873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:21.372658 21873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:21.376920 21873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:21.377144 21873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:21.395458 21873 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40053
I20260924 19:05:21.395504 21939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40053 every 8 connection(s)
I20260924 19:05:21.396744 21873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:05:21.399106 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 21873
I20260924 19:05:21.399325 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:05:21.399914 388 external_mini_cluster.cc:1428] Setting key 03876ff2a96bd6e70f1a3120f01605b6
I20260924 19:05:21.401261 21940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:05:21.413470 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:21.401785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.207:33589 (local address 127.0.0.1:40053)
0924 19:05:21.402537 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:21.402548 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:21.402589 (+ 41us) server_negotiation.cc:407] Connection header received
0924 19:05:21.404123 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:21.404142 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:21.404540 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:21.404861 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:21.405492 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:21.406821 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:21.408891 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:21.409609 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:21.409733 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:21.411179 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:21.411186 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:21.411199 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:05:21.411736 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:21.411806 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:05:21.411837 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:21.411965 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:21.412314 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 19:05:21.412517 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":341,"threads_started":1}
I20260924 19:05:21.415575 21940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48eeccb82f3e4764a4bf6f2bd18d9a47: Bootstrap starting.
I20260924 19:05:21.417392 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:05:21.419071 21940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48eeccb82f3e4764a4bf6f2bd18d9a47: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:21.420240 21940 log.cc:824] T 00000000000000000000000000000000 P 48eeccb82f3e4764a4bf6f2bd18d9a47: Log is configured to *not* fsync() on all Append() calls
I20260924 19:05:21.423214 21940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48eeccb82f3e4764a4bf6f2bd18d9a47: No bootstrap required, opened a new log
I20260924 19:05:21.428399 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:21.421643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.207:54453 (local address 127.0.0.1:40053)
0924 19:05:21.421830 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:21.421838 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:21.422285 (+ 447us) server_negotiation.cc:407] Connection header received
0924 19:05:21.422518 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:21.422522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:21.422580 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:21.422689 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:21.423204 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:21.424102 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:21.425619 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:21.426153 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:21.426322 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:21.427630 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:21.427635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:21.427638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:05:21.427655 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:21.427731 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:05:21.427750 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:21.427900 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:21.428083 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 19:05:21.428139 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 19:05:21.431459 21940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48eeccb82f3e4764a4bf6f2bd18d9a47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48eeccb82f3e4764a4bf6f2bd18d9a47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40053 } }
I20260924 19:05:21.431829 21940 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48eeccb82f3e4764a4bf6f2bd18d9a47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:05:21.431928 21940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48eeccb82f3e4764a4bf6f2bd18d9a47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48eeccb82f3e4764a4bf6f2bd18d9a47, State: Initialized, Role: FOLLOWER
I20260924 19:05:21.432479 21940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48eeccb82f3e4764a4bf6f2bd18d9a47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48eeccb82f3e4764a4bf6f2bd18d9a47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40053 } }
I20260924 19:05:21.432679 21940 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48eeccb82f3e4764a4bf6f2bd18d9a47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:05:21.432826 21940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48eeccb82f3e4764a4bf6f2bd18d9a47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:05:21.433003 21940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48eeccb82f3e4764a4bf6f2bd18d9a47 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:21.435307 21940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48eeccb82f3e4764a4bf6f2bd18d9a47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48eeccb82f3e4764a4bf6f2bd18d9a47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40053 } }
I20260924 19:05:21.435891 21940 leader_election.cc:302] T 00000000000000000000000000000000 P 48eeccb82f3e4764a4bf6f2bd18d9a47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48eeccb82f3e4764a4bf6f2bd18d9a47; no voters:
I20260924 19:05:21.436574 21940 leader_election.cc:288] T 00000000000000000000000000000000 P 48eeccb82f3e4764a4bf6f2bd18d9a47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:05:21.436694 21950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48eeccb82f3e4764a4bf6f2bd18d9a47 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:05:21.437361 21950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48eeccb82f3e4764a4bf6f2bd18d9a47 [term 1 LEADER]: Becoming Leader. State: Replica: 48eeccb82f3e4764a4bf6f2bd18d9a47, State: Running, Role: LEADER
I20260924 19:05:21.437981 21950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48eeccb82f3e4764a4bf6f2bd18d9a47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48eeccb82f3e4764a4bf6f2bd18d9a47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40053 } }
I20260924 19:05:21.438920 21940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48eeccb82f3e4764a4bf6f2bd18d9a47 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:05:21.440884 21951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48eeccb82f3e4764a4bf6f2bd18d9a47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48eeccb82f3e4764a4bf6f2bd18d9a47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48eeccb82f3e4764a4bf6f2bd18d9a47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40053 } } }
I20260924 19:05:21.441020 21952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48eeccb82f3e4764a4bf6f2bd18d9a47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48eeccb82f3e4764a4bf6f2bd18d9a47. Latest consensus state: current_term: 1 leader_uuid: "48eeccb82f3e4764a4bf6f2bd18d9a47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48eeccb82f3e4764a4bf6f2bd18d9a47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40053 } } }
I20260924 19:05:21.441386 21952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48eeccb82f3e4764a4bf6f2bd18d9a47 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:21.441419 21951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48eeccb82f3e4764a4bf6f2bd18d9a47 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:21.444615 21957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:05:21.449918 21957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:05:21.459082 21957 catalog_manager.cc:1347] Generated new cluster ID: 8d6d1fea754e4b5c9e968b8479e1b6b4
I20260924 19:05:21.459307 21957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:05:21.481570 21957 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:05:21.482841 21957 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:05:21.492936 21957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48eeccb82f3e4764a4bf6f2bd18d9a47: Generated new TSK 0
I20260924 19:05:21.493788 21957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:05:21.506647 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 21873
W20260924 19:05:21.507681 21935 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.207:33589
W20260924 19:05:21.508440 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 21873: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:05:21.508586 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 21873
W20260924 19:05:21.522300 21858 connection.cc:535] client connection to 127.0.0.1:40053 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40053): Connection reset by peer (error 104)
2026-09-24T19:05:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1894 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:05:21 dist-test-slave-9fn1 krb5kdc[21971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:05:21 dist-test-slave-9fn1 krb5kdc[21971](info): set up 2 sockets
Sep 24 19:05:21 dist-test-slave-9fn1 krb5kdc[21971](info): commencing operation
krb5kdc: starting...
W20260924 19:05:22.629844 388 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:05:22 dist-test-slave-9fn1 krb5kdc[21971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:05:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:05:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:05:22.868544 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39729
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:23.187866 21984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:23.188218 21984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:23.188380 21984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:23.195989 21984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:05:23.196146 21984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:23.196247 21984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:23.196362 21984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:05:23.196441 21984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:05:23.206084 21984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39729
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276386554690-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:23.207314 21984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:23.208577 21984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:23.220273 21989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:23.220311 21990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:23.220224 21992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:23.221048 21984 server_base.cc:1034] running on GCE node
I20260924 19:05:23.221756 21984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:23.223023 21984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:23.224220 21984 hybrid_clock.cc:648] HybridClock initialized: now 1790276723224185 us; error 25 us; skew 500 ppm
I20260924 19:05:23.224562 21984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:23 dist-test-slave-9fn1 krb5kdc[21971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:23.232079 21984 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:05:23.233462 21984 webserver.cc:460] Webserver started at http://127.0.0.1:34303/ using document root <none> and password file <none>
I20260924 19:05:23.234182 21984 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:23.234309 21984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:23.234630 21984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:23.237557 21984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "a2a127249d14471d930ef176814b6843"
format_stamp: "Formatted at 2026-09-24 19:05:23 on dist-test-slave-9fn1"
server_key: "90a504bb267e9009ef3c40168d7ff875"
server_key_iv: "a6a341165c4bd3e874e839635181b9ca"
server_key_version: "encryptionkey@0"
I20260924 19:05:23.238286 21984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "a2a127249d14471d930ef176814b6843"
format_stamp: "Formatted at 2026-09-24 19:05:23 on dist-test-slave-9fn1"
server_key: "90a504bb267e9009ef3c40168d7ff875"
server_key_iv: "a6a341165c4bd3e874e839635181b9ca"
server_key_version: "encryptionkey@0"
I20260924 19:05:23.243006 21984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 19:05:23.246170 21999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:23.247406 21984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 19:05:23.247572 21984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "a2a127249d14471d930ef176814b6843"
format_stamp: "Formatted at 2026-09-24 19:05:23 on dist-test-slave-9fn1"
server_key: "90a504bb267e9009ef3c40168d7ff875"
server_key_iv: "a6a341165c4bd3e874e839635181b9ca"
server_key_version: "encryptionkey@0"
I20260924 19:05:23.247745 21984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:23.253371 21984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:23.257303 21984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:23.257671 21984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:23.275342 21984 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39729
I20260924 19:05:23.275420 22050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39729 every 8 connection(s)
I20260924 19:05:23.276675 21984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:05:23.280171 22051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:05:23.280759 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 21984
I20260924 19:05:23.280931 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:05:23.281457 388 external_mini_cluster.cc:1428] Setting key ba8f2e910c54ba23c5166a3ca755d25f
I20260924 19:05:23.289366 22054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:23.283196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.207:42277 (local address 127.0.0.1:39729)
0924 19:05:23.283999 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:23.284013 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:23.284056 (+ 43us) server_negotiation.cc:407] Connection header received
0924 19:05:23.286212 (+ 2156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:23.286233 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:23.286561 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:23.286868 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:23.287916 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:23.287922 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:23.287934 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:05:23.287957 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:23.288011 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:05:23.288033 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:23.288188 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:23.288312 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 19:05:23.288498 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":374,"threads_started":1}
I20260924 19:05:23.292986 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:05:23.293390 22051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2a127249d14471d930ef176814b6843: Bootstrap starting.
I20260924 19:05:23.296259 22051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2a127249d14471d930ef176814b6843: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:23.297365 22051 log.cc:824] T 00000000000000000000000000000000 P a2a127249d14471d930ef176814b6843: Log is configured to *not* fsync() on all Append() calls
I20260924 19:05:23.299669 22054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:23.297158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.207:57363 (local address 127.0.0.1:39729)
0924 19:05:23.297326 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:23.297331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:23.297809 (+ 478us) server_negotiation.cc:407] Connection header received
0924 19:05:23.298000 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:23.298012 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:23.298058 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:23.298135 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:23.299046 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:23.299051 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:23.299055 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:05:23.299077 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:23.299117 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:05:23.299136 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:23.299251 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:23.299330 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 19:05:23.299384 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 19:05:23.300374 22051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2a127249d14471d930ef176814b6843: No bootstrap required, opened a new log
I20260924 19:05:23.307691 22051 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2a127249d14471d930ef176814b6843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a127249d14471d930ef176814b6843" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } }
I20260924 19:05:23.308068 22051 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2a127249d14471d930ef176814b6843 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:05:23.308189 22051 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2a127249d14471d930ef176814b6843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a127249d14471d930ef176814b6843, State: Initialized, Role: FOLLOWER
I20260924 19:05:23.308718 22051 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2a127249d14471d930ef176814b6843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a127249d14471d930ef176814b6843" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } }
I20260924 19:05:23.308970 22051 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2a127249d14471d930ef176814b6843 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:05:23.309106 22051 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2a127249d14471d930ef176814b6843 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:05:23.309314 22051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2a127249d14471d930ef176814b6843 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:23.311667 22051 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2a127249d14471d930ef176814b6843 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a127249d14471d930ef176814b6843" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } }
I20260924 19:05:23.312088 22051 leader_election.cc:302] T 00000000000000000000000000000000 P a2a127249d14471d930ef176814b6843 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2a127249d14471d930ef176814b6843; no voters:
I20260924 19:05:23.312589 22051 leader_election.cc:288] T 00000000000000000000000000000000 P a2a127249d14471d930ef176814b6843 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:05:23.312742 22061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2a127249d14471d930ef176814b6843 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:05:23.313364 22061 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2a127249d14471d930ef176814b6843 [term 1 LEADER]: Becoming Leader. State: Replica: a2a127249d14471d930ef176814b6843, State: Running, Role: LEADER
I20260924 19:05:23.313947 22061 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2a127249d14471d930ef176814b6843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a127249d14471d930ef176814b6843" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } }
I20260924 19:05:23.314502 22051 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2a127249d14471d930ef176814b6843 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:05:23.316820 22062 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2a127249d14471d930ef176814b6843 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2a127249d14471d930ef176814b6843" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a127249d14471d930ef176814b6843" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } } }
I20260924 19:05:23.317238 22062 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2a127249d14471d930ef176814b6843 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:23.317706 22063 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2a127249d14471d930ef176814b6843 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2a127249d14471d930ef176814b6843. Latest consensus state: current_term: 1 leader_uuid: "a2a127249d14471d930ef176814b6843" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2a127249d14471d930ef176814b6843" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39729 } } }
I20260924 19:05:23.318087 22063 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2a127249d14471d930ef176814b6843 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:23.319440 22072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:05:23.324213 22072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:05:23.332438 22072 catalog_manager.cc:1347] Generated new cluster ID: 095f377c793d45d789e4536e8baea3b7
I20260924 19:05:23.332538 22072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:05:23.363512 22072 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:05:23.363979 22072 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:05:23.375373 22072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2a127249d14471d930ef176814b6843: Generated new TSK 0
I20260924 19:05:23.376325 22072 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:05:23.392597 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 21984
W20260924 19:05:23.393724 22046 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.207:42277
W20260924 19:05:23.394419 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 21984: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:05:23.394562 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 21984
2026-09-24T19:05:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1885 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:05:23 dist-test-slave-9fn1 krb5kdc[22082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:05:23 dist-test-slave-9fn1 krb5kdc[22082](info): set up 2 sockets
Sep 24 19:05:23 dist-test-slave-9fn1 krb5kdc[22082](info): commencing operation
krb5kdc: starting...
W20260924 19:05:24.485925 388 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:05:24 dist-test-slave-9fn1 krb5kdc[22082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:05:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:05:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 19:05:24.736553 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34271
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34271
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:25.059351 22095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:25.059631 22095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:25.059824 22095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:25.067018 22095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:05:25.067162 22095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:25.067286 22095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:25.067397 22095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:05:25.067478 22095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:05:25.077170 22095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34271
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:25.078447 22095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:25.079679 22095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:25.091099 22100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:25.091020 22103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:25.091075 22101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:25.092550 22095 server_base.cc:1034] running on GCE node
I20260924 19:05:25.093554 22095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:25.095204 22095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:25.096403 22095 hybrid_clock.cc:648] HybridClock initialized: now 1790276725096362 us; error 34 us; skew 500 ppm
I20260924 19:05:25.096709 22095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:25 dist-test-slave-9fn1 krb5kdc[22082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:25.099740 22095 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:05:25.101227 22095 webserver.cc:460] Webserver started at http://127.0.97.62:34733/ using document root <none> and password file <none>
I20260924 19:05:25.101949 22095 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:25.102053 22095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:25.102303 22095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:25.105156 22095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "74f2aa871be84f198fa209bccba1b9e1"
format_stamp: "Formatted at 2026-09-24 19:05:25 on dist-test-slave-9fn1"
server_key: "540727f8e067ea225bc9cc30e113d719"
server_key_iv: "88e2852c83b31f4eb35882d5bfd1c5d5"
server_key_version: "encryptionkey@0"
I20260924 19:05:25.105959 22095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "74f2aa871be84f198fa209bccba1b9e1"
format_stamp: "Formatted at 2026-09-24 19:05:25 on dist-test-slave-9fn1"
server_key: "540727f8e067ea225bc9cc30e113d719"
server_key_iv: "88e2852c83b31f4eb35882d5bfd1c5d5"
server_key_version: "encryptionkey@0"
I20260924 19:05:25.110996 22095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260924 19:05:25.114289 22110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:25.115592 22095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:05:25.115729 22095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "74f2aa871be84f198fa209bccba1b9e1"
format_stamp: "Formatted at 2026-09-24 19:05:25 on dist-test-slave-9fn1"
server_key: "540727f8e067ea225bc9cc30e113d719"
server_key_iv: "88e2852c83b31f4eb35882d5bfd1c5d5"
server_key_version: "encryptionkey@0"
I20260924 19:05:25.115888 22095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:25.140558 22095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:25.144438 22095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:25.144722 22095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:25.162732 22095 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34271
I20260924 19:05:25.162765 22161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34271 every 8 connection(s)
I20260924 19:05:25.164079 22095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:05:25.167654 22162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:05:25.171578 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 22095
I20260924 19:05:25.171762 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:05:25.172295 388 external_mini_cluster.cc:1428] Setting key 7e2d0dd2ca4dc00871e3e61acb39fd33
I20260924 19:05:25.180877 22162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f2aa871be84f198fa209bccba1b9e1: Bootstrap starting.
I20260924 19:05:25.184376 22162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74f2aa871be84f198fa209bccba1b9e1: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:25.185632 22162 log.cc:824] T 00000000000000000000000000000000 P 74f2aa871be84f198fa209bccba1b9e1: Log is configured to *not* fsync() on all Append() calls
I20260924 19:05:25.188606 22162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74f2aa871be84f198fa209bccba1b9e1: No bootstrap required, opened a new log
Sep 24 19:05:25 dist-test-slave-9fn1 krb5kdc[22082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:05:25.198370 22162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74f2aa871be84f198fa209bccba1b9e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f2aa871be84f198fa209bccba1b9e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34271 } }
I20260924 19:05:25.198814 22162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74f2aa871be84f198fa209bccba1b9e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:05:25.198930 22162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74f2aa871be84f198fa209bccba1b9e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74f2aa871be84f198fa209bccba1b9e1, State: Initialized, Role: FOLLOWER
I20260924 19:05:25.199651 22162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74f2aa871be84f198fa209bccba1b9e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f2aa871be84f198fa209bccba1b9e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34271 } }
I20260924 19:05:25.199946 22162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74f2aa871be84f198fa209bccba1b9e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:05:25.200096 22162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74f2aa871be84f198fa209bccba1b9e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:05:25.200287 22162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74f2aa871be84f198fa209bccba1b9e1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:25.201990 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:25.179808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56133 (local address 127.0.97.62:34271)
0924 19:05:25.180615 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:25.180627 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:25.180666 (+ 39us) server_negotiation.cc:407] Connection header received
0924 19:05:25.182532 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:25.182559 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:25.182970 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:25.183281 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:25.185068 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:25.186520 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:25.187837 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:25.188850 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:25.189035 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:25.193143 (+ 4108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:25.193158 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:25.193177 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:25.193224 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:25.197243 (+ 4019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:25.198050 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:25.198057 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:25.198065 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:25.198261 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:25.198711 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:25.198716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:25.198720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:25.199444 (+ 724us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:25.199752 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:25.200218 (+ 466us) server_negotiation.cc:299] Negotiation successful
0924 19:05:25.200591 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":383,"threads_started":1}
I20260924 19:05:25.203496 22162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74f2aa871be84f198fa209bccba1b9e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f2aa871be84f198fa209bccba1b9e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34271 } }
I20260924 19:05:25.204049 22162 leader_election.cc:302] T 00000000000000000000000000000000 P 74f2aa871be84f198fa209bccba1b9e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74f2aa871be84f198fa209bccba1b9e1; no voters:
I20260924 19:05:25.204694 22162 leader_election.cc:288] T 00000000000000000000000000000000 P 74f2aa871be84f198fa209bccba1b9e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:05:25.204802 22167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74f2aa871be84f198fa209bccba1b9e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:05:25.206166 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:05:25.206393 22167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74f2aa871be84f198fa209bccba1b9e1 [term 1 LEADER]: Becoming Leader. State: Replica: 74f2aa871be84f198fa209bccba1b9e1, State: Running, Role: LEADER
I20260924 19:05:25.206895 22162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74f2aa871be84f198fa209bccba1b9e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:05:25.206874 22167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74f2aa871be84f198fa209bccba1b9e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f2aa871be84f198fa209bccba1b9e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34271 } }
I20260924 19:05:25.211401 22168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74f2aa871be84f198fa209bccba1b9e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74f2aa871be84f198fa209bccba1b9e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f2aa871be84f198fa209bccba1b9e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34271 } } }
I20260924 19:05:25.212050 22168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74f2aa871be84f198fa209bccba1b9e1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:25.211637 22169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74f2aa871be84f198fa209bccba1b9e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74f2aa871be84f198fa209bccba1b9e1. Latest consensus state: current_term: 1 leader_uuid: "74f2aa871be84f198fa209bccba1b9e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74f2aa871be84f198fa209bccba1b9e1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34271 } } }
I20260924 19:05:25.212472 22169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74f2aa871be84f198fa209bccba1b9e1 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:25.222723 22179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:05:25.228924 22179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:05:25.240660 22179 catalog_manager.cc:1347] Generated new cluster ID: 8031bdee5b3f4eee9c67f28bcf21ff77
I20260924 19:05:25.240772 22179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:05:25.242693 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:25.224256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36349 (local address 127.0.97.62:34271)
0924 19:05:25.224450 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:25.224458 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:25.224540 (+ 82us) server_negotiation.cc:407] Connection header received
0924 19:05:25.224851 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:25.224856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:25.224952 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:25.225073 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:25.227022 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:25.227833 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:25.229735 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:25.230680 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:25.230806 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:25.235089 (+ 4283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:25.235095 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:25.235100 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:25.235136 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:25.239286 (+ 4150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:25.240407 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:25.240412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:25.240416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:25.240554 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:25.241142 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:25.241146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:25.241148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:25.241453 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:25.241761 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:25.242167 (+ 406us) server_negotiation.cc:299] Negotiation successful
0924 19:05:25.242338 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 19:05:25.274899 22179 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:05:25.276788 22179 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:05:25.293486 22179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74f2aa871be84f198fa209bccba1b9e1: Generated new TSK 0
I20260924 19:05:25.294328 22179 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:05:25.302444 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 22095
I20260924 19:05:25.303717 22157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:05:25.395149 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 22095
W20260924 19:05:25.412161 22080 connection.cc:535] client connection to 127.0.97.62:34271 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:34271): Connection reset by peer (error 104)
2026-09-24T19:05:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2004 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:05:25 dist-test-slave-9fn1 krb5kdc[22194](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:05:25 dist-test-slave-9fn1 krb5kdc[22194](info): set up 2 sockets
Sep 24 19:05:25 dist-test-slave-9fn1 krb5kdc[22194](info): commencing operation
krb5kdc: starting...
W20260924 19:05:26.490438 388 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:05:26 dist-test-slave-9fn1 krb5kdc[22194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:05:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:05:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 19:05:26.736671 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37663
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:27.058473 22207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:27.058808 22207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:27.058988 22207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:27.065919 22207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:05:27.066078 22207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:27.066185 22207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:27.066293 22207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:05:27.066380 22207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:05:27.075640 22207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37663
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:27.076947 22207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:27.078263 22207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:27.089843 22212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:27.089890 22213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:27.090042 22215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:27.091341 22207 server_base.cc:1034] running on GCE node
I20260924 19:05:27.092124 22207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:27.093608 22207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:27.094823 22207 hybrid_clock.cc:648] HybridClock initialized: now 1790276727094778 us; error 39 us; skew 500 ppm
I20260924 19:05:27.095178 22207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:27 dist-test-slave-9fn1 krb5kdc[22194](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:27.098309 22207 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:05:27.100008 22207 webserver.cc:460] Webserver started at http://127.0.97.62:38251/ using document root <none> and password file <none>
I20260924 19:05:27.100893 22207 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:27.101055 22207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:27.101413 22207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:27.104642 22207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "c3518cbdb7aa450cbdb8307d505cbd80"
format_stamp: "Formatted at 2026-09-24 19:05:27 on dist-test-slave-9fn1"
server_key: "45ec485485894c9740d054537857bc78"
server_key_iv: "4d61c4c2de95af61014f507314ac5ab7"
server_key_version: "encryptionkey@0"
I20260924 19:05:27.105321 22207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "c3518cbdb7aa450cbdb8307d505cbd80"
format_stamp: "Formatted at 2026-09-24 19:05:27 on dist-test-slave-9fn1"
server_key: "45ec485485894c9740d054537857bc78"
server_key_iv: "4d61c4c2de95af61014f507314ac5ab7"
server_key_version: "encryptionkey@0"
I20260924 19:05:27.110350 22207 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260924 19:05:27.113668 22222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:27.114940 22207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:05:27.115132 22207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "c3518cbdb7aa450cbdb8307d505cbd80"
format_stamp: "Formatted at 2026-09-24 19:05:27 on dist-test-slave-9fn1"
server_key: "45ec485485894c9740d054537857bc78"
server_key_iv: "4d61c4c2de95af61014f507314ac5ab7"
server_key_version: "encryptionkey@0"
I20260924 19:05:27.115324 22207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:27.136184 22207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:27.140158 22207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:27.140465 22207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:27.158527 22207 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37663
I20260924 19:05:27.158558 22273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37663 every 8 connection(s)
I20260924 19:05:27.159895 22207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:05:27.160132 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 22207
I20260924 19:05:27.160318 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:05:27.160985 388 external_mini_cluster.cc:1428] Setting key 6fc6627eafa366bd6afa7e79527d9652
I20260924 19:05:27.164595 22274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:05:27.175928 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:27.162775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45341 (local address 127.0.97.62:37663)
0924 19:05:27.163564 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:27.163578 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:27.163626 (+ 48us) server_negotiation.cc:407] Connection header received
0924 19:05:27.165984 (+ 2358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:27.166010 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:27.166474 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:27.166877 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:27.167399 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:27.169376 (+ 1977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:27.170780 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:27.171784 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:27.171952 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:27.173093 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:27.173105 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:27.173122 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:05:27.173859 (+ 737us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:27.173989 (+ 130us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:05:27.174032 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:27.174208 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:27.174447 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 19:05:27.174629 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":334,"threads_started":1}
I20260924 19:05:27.178005 22274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3518cbdb7aa450cbdb8307d505cbd80: Bootstrap starting.
I20260924 19:05:27.179610 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:05:27.181540 22274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3518cbdb7aa450cbdb8307d505cbd80: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:27.182806 22274 log.cc:824] T 00000000000000000000000000000000 P c3518cbdb7aa450cbdb8307d505cbd80: Log is configured to *not* fsync() on all Append() calls
I20260924 19:05:27.186012 22274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3518cbdb7aa450cbdb8307d505cbd80: No bootstrap required, opened a new log
I20260924 19:05:27.191572 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:27.184899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46931 (local address 127.0.97.62:37663)
0924 19:05:27.185074 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:27.185090 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:27.185452 (+ 362us) server_negotiation.cc:407] Connection header received
0924 19:05:27.185665 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:27.185670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:27.185741 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:27.185840 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:27.186328 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:27.187478 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:27.188803 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:27.189494 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:27.189648 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:27.190763 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:27.190772 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:27.190779 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:05:27.190804 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:27.190853 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:05:27.190873 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:27.190991 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:27.191158 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 19:05:27.191214 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 19:05:27.195046 22274 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3518cbdb7aa450cbdb8307d505cbd80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3518cbdb7aa450cbdb8307d505cbd80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37663 } }
I20260924 19:05:27.195405 22274 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3518cbdb7aa450cbdb8307d505cbd80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:05:27.195487 22274 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3518cbdb7aa450cbdb8307d505cbd80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3518cbdb7aa450cbdb8307d505cbd80, State: Initialized, Role: FOLLOWER
I20260924 19:05:27.196017 22274 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3518cbdb7aa450cbdb8307d505cbd80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3518cbdb7aa450cbdb8307d505cbd80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37663 } }
I20260924 19:05:27.196210 22274 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3518cbdb7aa450cbdb8307d505cbd80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:05:27.196309 22274 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3518cbdb7aa450cbdb8307d505cbd80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:05:27.196436 22274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3518cbdb7aa450cbdb8307d505cbd80 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:27.198860 22274 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3518cbdb7aa450cbdb8307d505cbd80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3518cbdb7aa450cbdb8307d505cbd80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37663 } }
I20260924 19:05:27.199404 22274 leader_election.cc:302] T 00000000000000000000000000000000 P c3518cbdb7aa450cbdb8307d505cbd80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3518cbdb7aa450cbdb8307d505cbd80; no voters:
I20260924 19:05:27.200089 22274 leader_election.cc:288] T 00000000000000000000000000000000 P c3518cbdb7aa450cbdb8307d505cbd80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:05:27.200230 22284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3518cbdb7aa450cbdb8307d505cbd80 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:05:27.201187 22284 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3518cbdb7aa450cbdb8307d505cbd80 [term 1 LEADER]: Becoming Leader. State: Replica: c3518cbdb7aa450cbdb8307d505cbd80, State: Running, Role: LEADER
I20260924 19:05:27.201829 22284 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3518cbdb7aa450cbdb8307d505cbd80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3518cbdb7aa450cbdb8307d505cbd80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37663 } }
I20260924 19:05:27.202587 22274 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3518cbdb7aa450cbdb8307d505cbd80 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:05:27.204823 22285 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3518cbdb7aa450cbdb8307d505cbd80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3518cbdb7aa450cbdb8307d505cbd80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3518cbdb7aa450cbdb8307d505cbd80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37663 } } }
I20260924 19:05:27.205680 22285 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3518cbdb7aa450cbdb8307d505cbd80 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:27.205327 22286 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3518cbdb7aa450cbdb8307d505cbd80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3518cbdb7aa450cbdb8307d505cbd80. Latest consensus state: current_term: 1 leader_uuid: "c3518cbdb7aa450cbdb8307d505cbd80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3518cbdb7aa450cbdb8307d505cbd80" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37663 } } }
I20260924 19:05:27.206158 22286 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3518cbdb7aa450cbdb8307d505cbd80 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:27.206532 22289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:05:27.212627 22289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:05:27.221701 22289 catalog_manager.cc:1347] Generated new cluster ID: 749f0842bb744b639ff17ce428ad1363
I20260924 19:05:27.221812 22289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:05:27.228956 22289 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:05:27.230232 22289 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:05:27.239346 22289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3518cbdb7aa450cbdb8307d505cbd80: Generated new TSK 0
I20260924 19:05:27.240151 22289 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:05:27.245296 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 22207
W20260924 19:05:27.246665 22269 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45341
W20260924 19:05:27.247419 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 22207: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:05:27.247547 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 22207
W20260924 19:05:27.264268 22192 connection.cc:535] client connection to 127.0.97.62:37663 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:37663): Connection reset by peer (error 104)
2026-09-24T19:05:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1852 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:05:27 dist-test-slave-9fn1 krb5kdc[22305](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:05:27 dist-test-slave-9fn1 krb5kdc[22305](info): set up 2 sockets
Sep 24 19:05:27 dist-test-slave-9fn1 krb5kdc[22305](info): commencing operation
krb5kdc: starting...
W20260924 19:05:28.338052 388 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:05:28 dist-test-slave-9fn1 krb5kdc[22305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276728, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:05:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:05:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 19:05:28.581051 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36041
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:28.905054 22318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:28.905376 22318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:28.905473 22318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:28.912786 22318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:05:28.912914 22318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:28.912958 22318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:28.913012 22318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:05:28.913048 22318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:05:28.923246 22318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36041
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:28.924499 22318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:28.925846 22318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:28.937750 22323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:28.937801 22326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:28.937824 22324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:28.938621 22318 server_base.cc:1034] running on GCE node
I20260924 19:05:28.939389 22318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:28.940769 22318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:28.941994 22318 hybrid_clock.cc:648] HybridClock initialized: now 1790276728941954 us; error 35 us; skew 500 ppm
I20260924 19:05:28.942335 22318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:28 dist-test-slave-9fn1 krb5kdc[22305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:28.948976 22318 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:05:28.950431 22318 webserver.cc:460] Webserver started at http://127.0.97.62:39339/ using document root <none> and password file <none>
I20260924 19:05:28.951124 22318 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:28.951269 22318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:28.951577 22318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:28.954555 22318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "d975e95d751d45278180a6d66ab378a7"
format_stamp: "Formatted at 2026-09-24 19:05:28 on dist-test-slave-9fn1"
server_key: "3561fedca6eb7bf4d61d0850b92a06b9"
server_key_iv: "85eb4966ccff67d58013c55d306e1ed8"
server_key_version: "encryptionkey@0"
I20260924 19:05:28.955374 22318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "d975e95d751d45278180a6d66ab378a7"
format_stamp: "Formatted at 2026-09-24 19:05:28 on dist-test-slave-9fn1"
server_key: "3561fedca6eb7bf4d61d0850b92a06b9"
server_key_iv: "85eb4966ccff67d58013c55d306e1ed8"
server_key_version: "encryptionkey@0"
I20260924 19:05:28.960340 22318 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260924 19:05:28.963742 22333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:28.965034 22318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 19:05:28.965212 22318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "d975e95d751d45278180a6d66ab378a7"
format_stamp: "Formatted at 2026-09-24 19:05:28 on dist-test-slave-9fn1"
server_key: "3561fedca6eb7bf4d61d0850b92a06b9"
server_key_iv: "85eb4966ccff67d58013c55d306e1ed8"
server_key_version: "encryptionkey@0"
I20260924 19:05:28.965432 22318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:28.970896 22318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:28.974577 22318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:28.974802 22318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:28.992525 22318 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36041
I20260924 19:05:28.992563 22384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36041 every 8 connection(s)
I20260924 19:05:28.993863 22318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:05:28.996158 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 22318
I20260924 19:05:28.996322 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:05:28.996775 388 external_mini_cluster.cc:1428] Setting key 1f4bd4f68cc151defc37227a93002c93
I20260924 19:05:28.997853 22385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:05:29.004609 22388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:28.998510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48057 (local address 127.0.97.62:36041)
0924 19:05:28.999332 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:28.999348 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:28.999649 (+ 301us) server_negotiation.cc:407] Connection header received
0924 19:05:29.001306 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:29.001324 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:29.001721 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:29.002075 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:29.003075 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:29.003081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:29.003093 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:05:29.003105 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:29.003173 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:05:29.003191 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:29.003426 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:29.003608 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 19:05:29.003785 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":349,"threads_started":1}
I20260924 19:05:29.008131 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:05:29.010294 22385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d975e95d751d45278180a6d66ab378a7: Bootstrap starting.
I20260924 19:05:29.014060 22385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d975e95d751d45278180a6d66ab378a7: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:29.014729 22388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:29.012243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54319 (local address 127.0.97.62:36041)
0924 19:05:29.012441 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:29.012450 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:29.012811 (+ 361us) server_negotiation.cc:407] Connection header received
0924 19:05:29.012977 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:29.012980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:29.013023 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:29.013090 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:29.014050 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:29.014054 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:29.014057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:05:29.014071 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:29.014106 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:05:29.014142 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:29.014283 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:29.014413 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 19:05:29.014473 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 19:05:29.015309 22385 log.cc:824] T 00000000000000000000000000000000 P d975e95d751d45278180a6d66ab378a7: Log is configured to *not* fsync() on all Append() calls
I20260924 19:05:29.017855 22385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d975e95d751d45278180a6d66ab378a7: No bootstrap required, opened a new log
I20260924 19:05:29.024564 22385 raft_consensus.cc:348] T 00000000000000000000000000000000 P d975e95d751d45278180a6d66ab378a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975e95d751d45278180a6d66ab378a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } }
I20260924 19:05:29.024917 22385 raft_consensus.cc:374] T 00000000000000000000000000000000 P d975e95d751d45278180a6d66ab378a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:05:29.025017 22385 raft_consensus.cc:729] T 00000000000000000000000000000000 P d975e95d751d45278180a6d66ab378a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d975e95d751d45278180a6d66ab378a7, State: Initialized, Role: FOLLOWER
I20260924 19:05:29.025566 22385 consensus_queue.cc:260] T 00000000000000000000000000000000 P d975e95d751d45278180a6d66ab378a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975e95d751d45278180a6d66ab378a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } }
I20260924 19:05:29.025786 22385 raft_consensus.cc:388] T 00000000000000000000000000000000 P d975e95d751d45278180a6d66ab378a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:05:29.025925 22385 raft_consensus.cc:482] T 00000000000000000000000000000000 P d975e95d751d45278180a6d66ab378a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:05:29.026094 22385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d975e95d751d45278180a6d66ab378a7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:29.028149 22385 raft_consensus.cc:504] T 00000000000000000000000000000000 P d975e95d751d45278180a6d66ab378a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975e95d751d45278180a6d66ab378a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } }
I20260924 19:05:29.028627 22385 leader_election.cc:302] T 00000000000000000000000000000000 P d975e95d751d45278180a6d66ab378a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d975e95d751d45278180a6d66ab378a7; no voters:
I20260924 19:05:29.029170 22385 leader_election.cc:288] T 00000000000000000000000000000000 P d975e95d751d45278180a6d66ab378a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:05:29.029296 22395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d975e95d751d45278180a6d66ab378a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:05:29.030007 22395 raft_consensus.cc:686] T 00000000000000000000000000000000 P d975e95d751d45278180a6d66ab378a7 [term 1 LEADER]: Becoming Leader. State: Replica: d975e95d751d45278180a6d66ab378a7, State: Running, Role: LEADER
I20260924 19:05:29.030507 22395 consensus_queue.cc:237] T 00000000000000000000000000000000 P d975e95d751d45278180a6d66ab378a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975e95d751d45278180a6d66ab378a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } }
I20260924 19:05:29.031140 22385 sys_catalog.cc:564] T 00000000000000000000000000000000 P d975e95d751d45278180a6d66ab378a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:05:29.033591 22397 sys_catalog.cc:455] T 00000000000000000000000000000000 P d975e95d751d45278180a6d66ab378a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d975e95d751d45278180a6d66ab378a7. Latest consensus state: current_term: 1 leader_uuid: "d975e95d751d45278180a6d66ab378a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975e95d751d45278180a6d66ab378a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } } }
I20260924 19:05:29.034044 22397 sys_catalog.cc:458] T 00000000000000000000000000000000 P d975e95d751d45278180a6d66ab378a7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:29.033469 22396 sys_catalog.cc:455] T 00000000000000000000000000000000 P d975e95d751d45278180a6d66ab378a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d975e95d751d45278180a6d66ab378a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d975e95d751d45278180a6d66ab378a7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36041 } } }
I20260924 19:05:29.034613 22396 sys_catalog.cc:458] T 00000000000000000000000000000000 P d975e95d751d45278180a6d66ab378a7 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:29.039119 22403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:05:29.045816 22403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:05:29.057430 22403 catalog_manager.cc:1347] Generated new cluster ID: 71210481591f4026b5f3d5cb22299c0b
I20260924 19:05:29.057562 22403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:05:29.072656 22403 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:05:29.073139 22403 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:05:29.086578 22403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d975e95d751d45278180a6d66ab378a7: Generated new TSK 0
I20260924 19:05:29.087602 22403 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:05:29.100651 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 22318
W20260924 19:05:29.101887 22380 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48057
W20260924 19:05:29.102731 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 22318: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:05:29.102846 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 22318
2026-09-24T19:05:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1856 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:05:29 dist-test-slave-9fn1 krb5kdc[22416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:05:29 dist-test-slave-9fn1 krb5kdc[22416](info): set up 2 sockets
Sep 24 19:05:29 dist-test-slave-9fn1 krb5kdc[22416](info): commencing operation
krb5kdc: starting...
W20260924 19:05:30.201965 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:05:30 dist-test-slave-9fn1 krb5kdc[22416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:05:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:05:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 19:05:30.445420 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40545
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:30.770340 22429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:30.770678 22429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:30.770854 22429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:30.778118 22429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:05:30.778258 22429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:30.778362 22429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:30.778465 22429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:05:30.778558 22429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:05:30.788269 22429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:30.789537 22429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:30.790835 22429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:30.802981 22434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:30.802938 22435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:30.803014 22437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:30.803844 22429 server_base.cc:1034] running on GCE node
I20260924 19:05:30.804636 22429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:30.806072 22429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:30.807317 22429 hybrid_clock.cc:648] HybridClock initialized: now 1790276730807266 us; error 44 us; skew 500 ppm
I20260924 19:05:30.807660 22429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:30 dist-test-slave-9fn1 krb5kdc[22416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276730, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:30.810638 22429 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:05:30.812096 22429 webserver.cc:460] Webserver started at http://127.0.97.62:42641/ using document root <none> and password file <none>
I20260924 19:05:30.812770 22429 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:30.812914 22429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:30.813210 22429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:30.816186 22429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "aa88769a762849cd9893bcd1953ba00a"
format_stamp: "Formatted at 2026-09-24 19:05:30 on dist-test-slave-9fn1"
server_key: "f639be0a7f859c8b26f601955b9e7963"
server_key_iv: "12cc1e6d751ab6c0bb57f71550acfd9c"
server_key_version: "encryptionkey@0"
I20260924 19:05:30.816962 22429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "aa88769a762849cd9893bcd1953ba00a"
format_stamp: "Formatted at 2026-09-24 19:05:30 on dist-test-slave-9fn1"
server_key: "f639be0a7f859c8b26f601955b9e7963"
server_key_iv: "12cc1e6d751ab6c0bb57f71550acfd9c"
server_key_version: "encryptionkey@0"
I20260924 19:05:30.822136 22429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 19:05:30.825412 22444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:30.826762 22429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:05:30.826951 22429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "aa88769a762849cd9893bcd1953ba00a"
format_stamp: "Formatted at 2026-09-24 19:05:30 on dist-test-slave-9fn1"
server_key: "f639be0a7f859c8b26f601955b9e7963"
server_key_iv: "12cc1e6d751ab6c0bb57f71550acfd9c"
server_key_version: "encryptionkey@0"
I20260924 19:05:30.827145 22429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:30.844391 22429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:30.848194 22429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:30.848479 22429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:30.866521 22429 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40545
I20260924 19:05:30.866554 22495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40545 every 8 connection(s)
I20260924 19:05:30.867882 22429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:05:30.869491 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 22429
I20260924 19:05:30.869697 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:05:30.870219 388 external_mini_cluster.cc:1428] Setting key dc13942055afb6a10cdc2bbf71b45349
I20260924 19:05:30.872352 22496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 19:05:30 dist-test-slave-9fn1 krb5kdc[22416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:05:30.885147 22496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa88769a762849cd9893bcd1953ba00a: Bootstrap starting.
I20260924 19:05:30.889341 22496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa88769a762849cd9893bcd1953ba00a: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:30.890607 22496 log.cc:824] T 00000000000000000000000000000000 P aa88769a762849cd9893bcd1953ba00a: Log is configured to *not* fsync() on all Append() calls
I20260924 19:05:30.893872 22499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:30.871927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53601 (local address 127.0.97.62:40545)
0924 19:05:30.872705 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:30.872716 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:30.872753 (+ 37us) server_negotiation.cc:407] Connection header received
0924 19:05:30.874353 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:30.874374 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:30.874742 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:30.875054 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:30.876713 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:30.878053 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:30.879837 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:30.880472 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:30.880589 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:30.885221 (+ 4632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:30.885234 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:30.885252 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:30.885298 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:30.889143 (+ 3845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:30.890005 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:30.890012 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:30.890020 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:30.890220 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:30.890567 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:30.890572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:30.890574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:30.891330 (+ 756us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:30.891548 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:30.892124 (+ 576us) server_negotiation.cc:299] Negotiation successful
0924 19:05:30.892536 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":355,"threads_started":1}
I20260924 19:05:30.893976 22496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa88769a762849cd9893bcd1953ba00a: No bootstrap required, opened a new log
I20260924 19:05:30.897478 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:05:30.902523 22496 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa88769a762849cd9893bcd1953ba00a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa88769a762849cd9893bcd1953ba00a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40545 } }
I20260924 19:05:30.902891 22496 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa88769a762849cd9893bcd1953ba00a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:05:30.902997 22496 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa88769a762849cd9893bcd1953ba00a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa88769a762849cd9893bcd1953ba00a, State: Initialized, Role: FOLLOWER
I20260924 19:05:30.903540 22496 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa88769a762849cd9893bcd1953ba00a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa88769a762849cd9893bcd1953ba00a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40545 } }
I20260924 19:05:30.903781 22496 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa88769a762849cd9893bcd1953ba00a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:05:30.903923 22496 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa88769a762849cd9893bcd1953ba00a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:05:30.904098 22496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa88769a762849cd9893bcd1953ba00a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:30.906361 22496 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa88769a762849cd9893bcd1953ba00a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa88769a762849cd9893bcd1953ba00a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40545 } }
I20260924 19:05:30.906895 22496 leader_election.cc:302] T 00000000000000000000000000000000 P aa88769a762849cd9893bcd1953ba00a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa88769a762849cd9893bcd1953ba00a; no voters:
I20260924 19:05:30.907537 22496 leader_election.cc:288] T 00000000000000000000000000000000 P aa88769a762849cd9893bcd1953ba00a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:05:30.907619 22506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa88769a762849cd9893bcd1953ba00a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:05:30.909749 22506 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa88769a762849cd9893bcd1953ba00a [term 1 LEADER]: Becoming Leader. State: Replica: aa88769a762849cd9893bcd1953ba00a, State: Running, Role: LEADER
I20260924 19:05:30.910183 22496 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa88769a762849cd9893bcd1953ba00a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:05:30.910404 22506 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa88769a762849cd9893bcd1953ba00a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa88769a762849cd9893bcd1953ba00a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40545 } }
I20260924 19:05:30.914669 22507 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa88769a762849cd9893bcd1953ba00a [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa88769a762849cd9893bcd1953ba00a. Latest consensus state: current_term: 1 leader_uuid: "aa88769a762849cd9893bcd1953ba00a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa88769a762849cd9893bcd1953ba00a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40545 } } }
I20260924 19:05:30.915135 22507 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa88769a762849cd9893bcd1953ba00a [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:30.918486 22499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:30.901572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43681 (local address 127.0.97.62:40545)
0924 19:05:30.901715 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:30.901736 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:30.902145 (+ 409us) server_negotiation.cc:407] Connection header received
0924 19:05:30.902338 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:30.902347 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:30.902402 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:30.902498 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:30.904227 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:30.904989 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:30.906321 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:30.906818 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:30.906959 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:30.909599 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:30.909607 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:30.909614 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:30.909661 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:30.913244 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:30.914193 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:30.914198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:30.914202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:30.914409 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:30.914900 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:30.914908 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:30.914914 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:30.915574 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:30.915975 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:30.917655 (+ 1680us) server_negotiation.cc:299] Negotiation successful
0924 19:05:30.917974 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 19:05:30.920792 22508 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa88769a762849cd9893bcd1953ba00a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa88769a762849cd9893bcd1953ba00a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa88769a762849cd9893bcd1953ba00a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40545 } } }
I20260924 19:05:30.921066 22508 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa88769a762849cd9893bcd1953ba00a [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:30.930164 22516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:05:30.937002 22516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:05:30.947510 22516 catalog_manager.cc:1347] Generated new cluster ID: 942d0326675a422e949deac5202c2894
I20260924 19:05:30.947641 22516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:05:30.978310 22516 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:05:30.980326 22516 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:05:30.993655 22516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa88769a762849cd9893bcd1953ba00a: Generated new TSK 0
I20260924 19:05:30.994766 22516 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:05:31.002101 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 22429
I20260924 19:05:31.003463 22491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:05:31.095379 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 22429
2026-09-24T19:05:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1991 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:05:31 dist-test-slave-9fn1 krb5kdc[22528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:05:31 dist-test-slave-9fn1 krb5kdc[22528](info): set up 2 sockets
Sep 24 19:05:31 dist-test-slave-9fn1 krb5kdc[22528](info): commencing operation
krb5kdc: starting...
W20260924 19:05:32.197983 388 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:05:32 dist-test-slave-9fn1 krb5kdc[22528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:05:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:05:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 19:05:32.434653 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34375
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:32.763200 22541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:32.763465 22541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:32.763554 22541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:32.771319 22541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:05:32.771422 22541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:32.771467 22541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:32.771519 22541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:05:32.771556 22541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:05:32.781075 22541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:32.782297 22541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:32.783489 22541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:32.795280 22546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:32.795321 22547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:32.795351 22549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:32.796404 22541 server_base.cc:1034] running on GCE node
I20260924 19:05:32.797135 22541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:32.798520 22541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:32.799721 22541 hybrid_clock.cc:648] HybridClock initialized: now 1790276732799674 us; error 33 us; skew 500 ppm
I20260924 19:05:32.800060 22541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:32 dist-test-slave-9fn1 krb5kdc[22528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:32.803076 22541 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:05:32.804558 22541 webserver.cc:460] Webserver started at http://127.0.97.62:40473/ using document root <none> and password file <none>
I20260924 19:05:32.805225 22541 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:32.805368 22541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:32.805729 22541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:32.808640 22541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "a153b45ba6a14b76bc54b285dd918ed5"
format_stamp: "Formatted at 2026-09-24 19:05:32 on dist-test-slave-9fn1"
server_key: "b1e98a4efc7ea6375898193c93449244"
server_key_iv: "ebdd26c57a37bd138dce03004de66663"
server_key_version: "encryptionkey@0"
I20260924 19:05:32.809392 22541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "a153b45ba6a14b76bc54b285dd918ed5"
format_stamp: "Formatted at 2026-09-24 19:05:32 on dist-test-slave-9fn1"
server_key: "b1e98a4efc7ea6375898193c93449244"
server_key_iv: "ebdd26c57a37bd138dce03004de66663"
server_key_version: "encryptionkey@0"
I20260924 19:05:32.814299 22541 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 19:05:32.817468 22556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:32.818825 22541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:05:32.819011 22541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "a153b45ba6a14b76bc54b285dd918ed5"
format_stamp: "Formatted at 2026-09-24 19:05:32 on dist-test-slave-9fn1"
server_key: "b1e98a4efc7ea6375898193c93449244"
server_key_iv: "ebdd26c57a37bd138dce03004de66663"
server_key_version: "encryptionkey@0"
I20260924 19:05:32.819211 22541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:32.833886 22541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:32.838044 22541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:32.838347 22541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:32.857064 22541 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34375
I20260924 19:05:32.857097 22607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34375 every 8 connection(s)
I20260924 19:05:32.858493 22541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:05:32.858824 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 22541
I20260924 19:05:32.858996 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:05:32.859493 388 external_mini_cluster.cc:1428] Setting key 9bc3a064d6548c1d72b23316b96eb86e
I20260924 19:05:32.862933 22608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:05:32.872267 22611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:32.861372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37139 (local address 127.0.97.62:34375)
0924 19:05:32.862142 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:32.862154 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:32.862194 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:05:32.863874 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:32.863895 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:32.864263 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:32.864653 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:32.865186 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:32.866567 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:32.867891 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:32.868565 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:32.868775 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:32.870050 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:32.870061 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:32.870075 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:05:32.870620 (+ 545us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:32.870691 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:05:32.870718 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:32.870859 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:32.871061 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 19:05:32.871321 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":335,"threads_started":1}
I20260924 19:05:32.874111 22608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a153b45ba6a14b76bc54b285dd918ed5: Bootstrap starting.
I20260924 19:05:32.875957 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:05:32.877322 22608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a153b45ba6a14b76bc54b285dd918ed5: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:32.878340 22608 log.cc:824] T 00000000000000000000000000000000 P a153b45ba6a14b76bc54b285dd918ed5: Log is configured to *not* fsync() on all Append() calls
I20260924 19:05:32.881256 22608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a153b45ba6a14b76bc54b285dd918ed5: No bootstrap required, opened a new log
I20260924 19:05:32.886193 22611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:32.880176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56483 (local address 127.0.97.62:34375)
0924 19:05:32.880410 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:32.880419 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:32.880732 (+ 313us) server_negotiation.cc:407] Connection header received
0924 19:05:32.880937 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:32.880942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:32.881007 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:32.881108 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:32.881705 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:32.882416 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:32.883752 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:32.884216 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:32.884373 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:32.885428 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:32.885433 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:32.885436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:05:32.885454 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:32.885491 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:05:32.885546 (+ 55us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:32.885632 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:32.885794 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 19:05:32.885885 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:05:32.888950 22608 raft_consensus.cc:348] T 00000000000000000000000000000000 P a153b45ba6a14b76bc54b285dd918ed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a153b45ba6a14b76bc54b285dd918ed5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34375 } }
I20260924 19:05:32.889333 22608 raft_consensus.cc:374] T 00000000000000000000000000000000 P a153b45ba6a14b76bc54b285dd918ed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:05:32.889438 22608 raft_consensus.cc:729] T 00000000000000000000000000000000 P a153b45ba6a14b76bc54b285dd918ed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a153b45ba6a14b76bc54b285dd918ed5, State: Initialized, Role: FOLLOWER
I20260924 19:05:32.890085 22608 consensus_queue.cc:260] T 00000000000000000000000000000000 P a153b45ba6a14b76bc54b285dd918ed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a153b45ba6a14b76bc54b285dd918ed5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34375 } }
I20260924 19:05:32.890331 22608 raft_consensus.cc:388] T 00000000000000000000000000000000 P a153b45ba6a14b76bc54b285dd918ed5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:05:32.890468 22608 raft_consensus.cc:482] T 00000000000000000000000000000000 P a153b45ba6a14b76bc54b285dd918ed5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:05:32.890643 22608 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a153b45ba6a14b76bc54b285dd918ed5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:32.893182 22608 raft_consensus.cc:504] T 00000000000000000000000000000000 P a153b45ba6a14b76bc54b285dd918ed5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a153b45ba6a14b76bc54b285dd918ed5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34375 } }
I20260924 19:05:32.893748 22608 leader_election.cc:302] T 00000000000000000000000000000000 P a153b45ba6a14b76bc54b285dd918ed5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a153b45ba6a14b76bc54b285dd918ed5; no voters:
I20260924 19:05:32.894354 22608 leader_election.cc:288] T 00000000000000000000000000000000 P a153b45ba6a14b76bc54b285dd918ed5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:05:32.894524 22618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a153b45ba6a14b76bc54b285dd918ed5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:05:32.895190 22618 raft_consensus.cc:686] T 00000000000000000000000000000000 P a153b45ba6a14b76bc54b285dd918ed5 [term 1 LEADER]: Becoming Leader. State: Replica: a153b45ba6a14b76bc54b285dd918ed5, State: Running, Role: LEADER
I20260924 19:05:32.895869 22618 consensus_queue.cc:237] T 00000000000000000000000000000000 P a153b45ba6a14b76bc54b285dd918ed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a153b45ba6a14b76bc54b285dd918ed5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34375 } }
I20260924 19:05:32.896605 22608 sys_catalog.cc:564] T 00000000000000000000000000000000 P a153b45ba6a14b76bc54b285dd918ed5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:05:32.898841 22620 sys_catalog.cc:455] T 00000000000000000000000000000000 P a153b45ba6a14b76bc54b285dd918ed5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a153b45ba6a14b76bc54b285dd918ed5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a153b45ba6a14b76bc54b285dd918ed5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34375 } } }
I20260924 19:05:32.899231 22620 sys_catalog.cc:458] T 00000000000000000000000000000000 P a153b45ba6a14b76bc54b285dd918ed5 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:32.898954 22619 sys_catalog.cc:455] T 00000000000000000000000000000000 P a153b45ba6a14b76bc54b285dd918ed5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a153b45ba6a14b76bc54b285dd918ed5. Latest consensus state: current_term: 1 leader_uuid: "a153b45ba6a14b76bc54b285dd918ed5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a153b45ba6a14b76bc54b285dd918ed5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34375 } } }
I20260924 19:05:32.899392 22619 sys_catalog.cc:458] T 00000000000000000000000000000000 P a153b45ba6a14b76bc54b285dd918ed5 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:32.902129 22624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:05:32.906914 22624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:05:32.915581 22624 catalog_manager.cc:1347] Generated new cluster ID: 4ac46843811c4755a2fe1a0d1a8f4d66
I20260924 19:05:32.915729 22624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:05:32.931591 22624 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:05:32.933538 22624 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:05:32.965487 22624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a153b45ba6a14b76bc54b285dd918ed5: Generated new TSK 0
I20260924 19:05:32.966388 22624 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:05:32.976649 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 22541
W20260924 19:05:32.977708 22603 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37139
W20260924 19:05:32.978389 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 22541: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:05:32.978518 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 22541
2026-09-24T19:05:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1880 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:05:33 dist-test-slave-9fn1 krb5kdc[22639](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:05:33 dist-test-slave-9fn1 krb5kdc[22639](info): set up 2 sockets
Sep 24 19:05:33 dist-test-slave-9fn1 krb5kdc[22639](info): commencing operation
krb5kdc: starting...
W20260924 19:05:34.058027 388 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.003s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:05:34 dist-test-slave-9fn1 krb5kdc[22639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:05:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:05:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 19:05:34.304113 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35341
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:34.628726 22652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:34.628998 22652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:34.629168 22652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:34.636351 22652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:05:34.636459 22652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:34.636504 22652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:34.636600 22652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:05:34.636688 22652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:05:34.646389 22652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:34.647627 22652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:34.648893 22652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:34.660374 22660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:34.660337 22657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:34.660579 22658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:34.661828 22652 server_base.cc:1034] running on GCE node
I20260924 19:05:34.662657 22652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:34.664139 22652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:34.665369 22652 hybrid_clock.cc:648] HybridClock initialized: now 1790276734665330 us; error 38 us; skew 500 ppm
I20260924 19:05:34.665782 22652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:34 dist-test-slave-9fn1 krb5kdc[22639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:34.672142 22652 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:05:34.673780 22652 webserver.cc:460] Webserver started at http://127.0.97.62:39141/ using document root <none> and password file <none>
I20260924 19:05:34.674474 22652 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:34.674634 22652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:34.674947 22652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:34.677945 22652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "8c08886c827a4d408685952bdc3954b5"
format_stamp: "Formatted at 2026-09-24 19:05:34 on dist-test-slave-9fn1"
server_key: "393024ebf12e2af1749b0488868defc9"
server_key_iv: "25caad01be8bda8cc1ba2b069db2e128"
server_key_version: "encryptionkey@0"
I20260924 19:05:34.678701 22652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "8c08886c827a4d408685952bdc3954b5"
format_stamp: "Formatted at 2026-09-24 19:05:34 on dist-test-slave-9fn1"
server_key: "393024ebf12e2af1749b0488868defc9"
server_key_iv: "25caad01be8bda8cc1ba2b069db2e128"
server_key_version: "encryptionkey@0"
I20260924 19:05:34.683734 22652 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:05:34.687196 22667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:34.688637 22652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:05:34.688848 22652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "8c08886c827a4d408685952bdc3954b5"
format_stamp: "Formatted at 2026-09-24 19:05:34 on dist-test-slave-9fn1"
server_key: "393024ebf12e2af1749b0488868defc9"
server_key_iv: "25caad01be8bda8cc1ba2b069db2e128"
server_key_version: "encryptionkey@0"
I20260924 19:05:34.689072 22652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:34.694854 22652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:34.698791 22652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:34.699081 22652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:34.717183 22652 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35341
I20260924 19:05:34.717229 22718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35341 every 8 connection(s)
I20260924 19:05:34.718566 22652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:05:34.722064 22719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:05:34.728650 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 22652
I20260924 19:05:34.728853 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:05:34.729362 388 external_mini_cluster.cc:1428] Setting key 131a0ec1db0400db5eb12ea2aca7c5e3
I20260924 19:05:34.732743 22719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c08886c827a4d408685952bdc3954b5: Bootstrap starting.
I20260924 19:05:34.736207 22719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c08886c827a4d408685952bdc3954b5: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:34.737056 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:34.731116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48259 (local address 127.0.97.62:35341)
0924 19:05:34.731974 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:34.731985 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:34.732026 (+ 41us) server_negotiation.cc:407] Connection header received
0924 19:05:34.733812 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:34.733834 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:34.734154 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:34.734441 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:34.735511 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:34.735517 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:34.735558 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:05:34.735572 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:34.735628 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:05:34.735645 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:34.735773 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:34.735943 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 19:05:34.736138 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":421,"threads_started":1}
I20260924 19:05:34.737272 22719 log.cc:824] T 00000000000000000000000000000000 P 8c08886c827a4d408685952bdc3954b5: Log is configured to *not* fsync() on all Append() calls
I20260924 19:05:34.740355 22719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c08886c827a4d408685952bdc3954b5: No bootstrap required, opened a new log
I20260924 19:05:34.741024 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:05:34.748515 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:34.746080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45873 (local address 127.0.97.62:35341)
0924 19:05:34.746263 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:34.746269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:34.746721 (+ 452us) server_negotiation.cc:407] Connection header received
0924 19:05:34.746846 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:34.746851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:34.746912 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:34.747019 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:34.747877 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:34.747884 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:34.747889 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:05:34.747907 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:34.747957 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:05:34.747981 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:34.748098 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:34.748163 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 19:05:34.748218 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 19:05:34.750290 22719 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c08886c827a4d408685952bdc3954b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c08886c827a4d408685952bdc3954b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35341 } }
I20260924 19:05:34.750624 22719 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c08886c827a4d408685952bdc3954b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:05:34.750741 22719 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c08886c827a4d408685952bdc3954b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c08886c827a4d408685952bdc3954b5, State: Initialized, Role: FOLLOWER
I20260924 19:05:34.751269 22719 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c08886c827a4d408685952bdc3954b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c08886c827a4d408685952bdc3954b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35341 } }
I20260924 19:05:34.751497 22719 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c08886c827a4d408685952bdc3954b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:05:34.751650 22719 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c08886c827a4d408685952bdc3954b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:05:34.751822 22719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c08886c827a4d408685952bdc3954b5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:34.754065 22719 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c08886c827a4d408685952bdc3954b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c08886c827a4d408685952bdc3954b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35341 } }
I20260924 19:05:34.754539 22719 leader_election.cc:302] T 00000000000000000000000000000000 P 8c08886c827a4d408685952bdc3954b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c08886c827a4d408685952bdc3954b5; no voters:
I20260924 19:05:34.755059 22719 leader_election.cc:288] T 00000000000000000000000000000000 P 8c08886c827a4d408685952bdc3954b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:05:34.755182 22729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c08886c827a4d408685952bdc3954b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:05:34.755863 22729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c08886c827a4d408685952bdc3954b5 [term 1 LEADER]: Becoming Leader. State: Replica: 8c08886c827a4d408685952bdc3954b5, State: Running, Role: LEADER
I20260924 19:05:34.756385 22729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c08886c827a4d408685952bdc3954b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c08886c827a4d408685952bdc3954b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35341 } }
I20260924 19:05:34.757094 22719 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c08886c827a4d408685952bdc3954b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:05:34.759370 22731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c08886c827a4d408685952bdc3954b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c08886c827a4d408685952bdc3954b5. Latest consensus state: current_term: 1 leader_uuid: "8c08886c827a4d408685952bdc3954b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c08886c827a4d408685952bdc3954b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35341 } } }
I20260924 19:05:34.759781 22731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c08886c827a4d408685952bdc3954b5 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:34.760260 22730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c08886c827a4d408685952bdc3954b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c08886c827a4d408685952bdc3954b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c08886c827a4d408685952bdc3954b5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35341 } } }
I20260924 19:05:34.760519 22730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c08886c827a4d408685952bdc3954b5 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:34.762230 22737 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:05:34.768433 22737 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:05:34.778744 22737 catalog_manager.cc:1347] Generated new cluster ID: c54535cb7cdc4e918ba48d1f4b831596
I20260924 19:05:34.778872 22737 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:05:34.792505 22737 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:05:34.792944 22737 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:05:34.803623 22737 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c08886c827a4d408685952bdc3954b5: Generated new TSK 0
I20260924 19:05:34.804410 22737 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:05:34.813484 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 22652
W20260924 19:05:34.814651 22714 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48259
W20260924 19:05:34.815346 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 22652: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:05:34.815495 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 22652
2026-09-24T19:05:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1839 ms)
[----------] 18 tests from AuthTokenIssuingTest (34383 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:05:34 dist-test-slave-9fn1 krb5kdc[22750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:05:34 dist-test-slave-9fn1 krb5kdc[22750](info): set up 2 sockets
Sep 24 19:05:34 dist-test-slave-9fn1 krb5kdc[22750](info): commencing operation
krb5kdc: starting...
W20260924 19:05:35.894098 388 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.010s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:05:36 dist-test-slave-9fn1 krb5kdc[22750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:05:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:05:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 19:05:36.142396 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42153
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42153,127.0.97.61:34697,127.0.97.60:39823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:36.462634 22763 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:05:36.462972 22763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:36.463104 22763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:36.463253 22763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:36.470255 22763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:05:36.470340 22763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:36.470441 22763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:36.470533 22763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:05:36.470628 22763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:05:36.480176 22763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42153,127.0.97.61:34697,127.0.97.60:39823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:36.481432 22763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:36.482715 22763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:36.494181 22768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:36.494153 22771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:36.494143 22769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:36.494956 22763 server_base.cc:1034] running on GCE node
I20260924 19:05:36.495698 22763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:36.497017 22763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:36.498245 22763 hybrid_clock.cc:648] HybridClock initialized: now 1790276736498190 us; error 35 us; skew 500 ppm
I20260924 19:05:36.498553 22763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:36 dist-test-slave-9fn1 krb5kdc[22750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:36.501484 22763 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:05:36.502861 22763 webserver.cc:460] Webserver started at http://127.0.97.62:37813/ using document root <none> and password file <none>
I20260924 19:05:36.503512 22763 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:36.503646 22763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:36.503962 22763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:36.506847 22763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "53ce32c546cb494cbe7111c9358a7c20"
format_stamp: "Formatted at 2026-09-24 19:05:36 on dist-test-slave-9fn1"
server_key: "f84ad11a0a5c46688ab366e6689adb0a"
server_key_iv: "03c62bb0544234ac6dbfc00449ecce50"
server_key_version: "encryptionkey@0"
I20260924 19:05:36.507613 22763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "53ce32c546cb494cbe7111c9358a7c20"
format_stamp: "Formatted at 2026-09-24 19:05:36 on dist-test-slave-9fn1"
server_key: "f84ad11a0a5c46688ab366e6689adb0a"
server_key_iv: "03c62bb0544234ac6dbfc00449ecce50"
server_key_version: "encryptionkey@0"
I20260924 19:05:36.512501 22763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260924 19:05:36.515843 22778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:36.517130 22763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 19:05:36.517303 22763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "53ce32c546cb494cbe7111c9358a7c20"
format_stamp: "Formatted at 2026-09-24 19:05:36 on dist-test-slave-9fn1"
server_key: "f84ad11a0a5c46688ab366e6689adb0a"
server_key_iv: "03c62bb0544234ac6dbfc00449ecce50"
server_key_version: "encryptionkey@0"
I20260924 19:05:36.517555 22763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:36.530126 22763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:36.533983 22763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:36.534260 22763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:36.553328 22763 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42153
I20260924 19:05:36.553303 22829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42153 every 8 connection(s)
I20260924 19:05:36.554796 22763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:05:36.555181 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 22763
I20260924 19:05:36.555377 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:05:36.556208 388 external_mini_cluster.cc:1428] Setting key d260fb3020766c42a0994ccc42b0f120
I20260924 19:05:36.558868 22830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:05:36.565445 22830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42153 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 24 19:05:36 dist-test-slave-9fn1 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:05:36.591912 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:36.568658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42153 (local address 127.0.0.1:59092)
0924 19:05:36.569839 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:36.569873 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:36.571627 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:36.572488 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:36.572494 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:36.573228 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:05:36.573634 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:36.573653 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:36.575198 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:36.575203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:36.576259 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:36.576270 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:36.577193 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:36.577199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:36.577365 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:36.578134 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:05:36.578156 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:05:36.582368 (+ 4212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:05:36.586907 (+ 4539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:36.586918 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:36.586939 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:36.587485 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:36.588009 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:36.588014 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:36.588017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:36.588251 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:36.589390 (+ 1139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:05:36.589400 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:05:36.589728 (+ 328us) client_negotiation.cc:769] Sending connection context
0924 19:05:36.590033 (+ 305us) client_negotiation.cc:240] Negotiation successful
0924 19:05:36.590461 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":982,"thread_start_us":779,"threads_started":1}
I20260924 19:05:36.593353 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:36.568657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59092 (local address 127.0.97.62:42153)
0924 19:05:36.569557 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:36.569576 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:36.570001 (+ 425us) server_negotiation.cc:407] Connection header received
0924 19:05:36.572012 (+ 2011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:36.572017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:36.572261 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:36.572472 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:36.573848 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:36.575026 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:36.576410 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:36.576950 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:36.577163 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:36.582670 (+ 5507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:36.582681 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:36.582700 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:36.582762 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:36.586739 (+ 3977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:36.587625 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:36.587650 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:36.587654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:36.587876 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:36.588392 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:36.588397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:36.588400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:36.589197 (+ 797us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:36.589404 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:36.592604 (+ 3200us) server_negotiation.cc:299] Negotiation successful
0924 19:05:36.592867 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":405,"threads_started":1}
I20260924 19:05:36.595935 22830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34697 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:05:36.597010 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:36.596530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34697 (local address 127.0.0.1:57512)
0924 19:05:36.596729 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:36.596863 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 19:05:36.597266 22782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34697: connect: Connection refused (error 111)
I20260924 19:05:36.598989 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:36.598535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34697 (local address 127.0.0.1:57526)
0924 19:05:36.598715 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:36.598804 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 19:05:36.599198 22830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34697: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34697: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260924 19:05:36.614545 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34697
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42153,127.0.97.61:34697,127.0.97.60:39823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:05:36.648542 22830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34697 } attempt: 1
I20260924 19:05:36.649729 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:36.649259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34697 (local address 127.0.0.1:57534)
0924 19:05:36.649446 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:36.649587 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 19:05:36.650739 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:36.650338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34697 (local address 127.0.0.1:57538)
0924 19:05:36.650482 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:36.650589 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 19:05:36.650995 22830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34697: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34697: connect: Connection refused (error 111)
I20260924 19:05:36.719260 22830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34697 } attempt: 2
I20260924 19:05:36.720395 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:36.719910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34697 (local address 127.0.0.1:57550)
0924 19:05:36.720173 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:36.720260 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 19:05:36.721457 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:36.721110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34697 (local address 127.0.0.1:57556)
0924 19:05:36.721261 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:36.721335 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 19:05:36.721735 22830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34697: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34697: connect: Connection refused (error 111)
I20260924 19:05:36.813036 22830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34697 } attempt: 3
I20260924 19:05:36.814345 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:36.813899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34697 (local address 127.0.0.1:57570)
0924 19:05:36.814111 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:36.814212 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 19:05:36.815508 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:36.815156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34697 (local address 127.0.0.1:57574)
0924 19:05:36.815317 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:36.815381 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260924 19:05:36.815761 22830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34697: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34697: connect: Connection refused (error 111)
W20260924 19:05:36.937585 22837 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:05:36.937892 22837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:36.937960 22837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:36.938038 22837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:36.944981 22837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:05:36.945070 22837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:36.945128 22837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:36.945183 22837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:05:36.945241 22837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:05:36.954921 22837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42153,127.0.97.61:34697,127.0.97.60:39823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:36.956243 22837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:36.957549 22837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:05:36.959002 22830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34697 } attempt: 4
I20260924 19:05:36.960245 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:36.959711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34697 (local address 127.0.0.1:57586)
0924 19:05:36.960025 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:36.960113 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260924 19:05:36.961310 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:36.960963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34697 (local address 127.0.0.1:57598)
0924 19:05:36.961110 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:36.961190 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 19:05:36.961597 22830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34697: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34697: connect: Connection refused (error 111)
W20260924 19:05:36.970271 22845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:36.970309 22843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:36.970367 22842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:36.971038 22837 server_base.cc:1034] running on GCE node
I20260924 19:05:36.971833 22837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:36.973194 22837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:36.974434 22837 hybrid_clock.cc:648] HybridClock initialized: now 1790276736974357 us; error 56 us; skew 500 ppm
I20260924 19:05:36.974758 22837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:36 dist-test-slave-9fn1 krb5kdc[22750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:36.977875 22837 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260924 19:05:36.979408 22837 webserver.cc:460] Webserver started at http://127.0.97.61:35367/ using document root <none> and password file <none>
I20260924 19:05:36.980090 22837 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:36.980225 22837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:36.980535 22837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:36.983456 22837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-1/data/instance:
uuid: "6d5fa1a56dc049f78281d8904f98af0a"
format_stamp: "Formatted at 2026-09-24 19:05:36 on dist-test-slave-9fn1"
server_key: "4744f2230e3e964198643b2c014976c4"
server_key_iv: "4e2c1df1befe4fec239ad92f8b642903"
server_key_version: "encryptionkey@0"
I20260924 19:05:36.984230 22837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-1/wal/instance:
uuid: "6d5fa1a56dc049f78281d8904f98af0a"
format_stamp: "Formatted at 2026-09-24 19:05:36 on dist-test-slave-9fn1"
server_key: "4744f2230e3e964198643b2c014976c4"
server_key_iv: "4e2c1df1befe4fec239ad92f8b642903"
server_key_version: "encryptionkey@0"
I20260924 19:05:36.989174 22837 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 19:05:36.992439 22852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:36.993734 22837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 19:05:36.993899 22837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-1/wal
uuid: "6d5fa1a56dc049f78281d8904f98af0a"
format_stamp: "Formatted at 2026-09-24 19:05:36 on dist-test-slave-9fn1"
server_key: "4744f2230e3e964198643b2c014976c4"
server_key_iv: "4e2c1df1befe4fec239ad92f8b642903"
server_key_version: "encryptionkey@0"
I20260924 19:05:36.994113 22837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:37.010417 22837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:37.014232 22837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:37.014503 22837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:37.032953 22837 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:34697
I20260924 19:05:37.032979 22903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:34697 every 8 connection(s)
I20260924 19:05:37.034323 22837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-1/data/info.pb
I20260924 19:05:37.037295 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 22837
I20260924 19:05:37.037477 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-1/wal/instance
I20260924 19:05:37.037892 388 external_mini_cluster.cc:1428] Setting key 6d6ed8092414bc6bb24e11062b635cee
I20260924 19:05:37.038043 22904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:05:37.044171 22904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42153 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:05:37 dist-test-slave-9fn1 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260924 19:05:37.064350 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.046788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59102 (local address 127.0.97.62:42153)
0924 19:05:37.046963 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:37.046967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:37.048116 (+ 1149us) server_negotiation.cc:407] Connection header received
0924 19:05:37.049961 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:37.049966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:37.050060 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:37.050175 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:37.051706 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.052379 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.053963 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.054370 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.054509 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:37.057938 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:37.057947 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:37.057953 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:37.057997 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:37.061074 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:37.061974 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:37.061984 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:37.061990 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:37.062138 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:37.062668 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:37.062671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:37.062673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:37.062937 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:37.063046 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:37.063953 (+ 907us) server_negotiation.cc:299] Negotiation successful
0924 19:05:37.064115 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:05:37.065817 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.047241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42153 (local address 127.0.0.1:59102)
0924 19:05:37.047963 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:37.048002 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:37.049702 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:37.050384 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:37.050392 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:37.050968 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:05:37.051510 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:37.051525 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.052539 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.052544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:37.053795 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:37.053811 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.054606 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.054613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:37.054844 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:37.055526 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:05:37.055560 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:05:37.057709 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:05:37.061238 (+ 3529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:37.061244 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:37.061257 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:37.061840 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:37.062267 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:37.062272 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:37.062275 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:37.062526 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:37.063081 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:05:37.063093 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:05:37.063565 (+ 472us) client_negotiation.cc:769] Sending connection context
0924 19:05:37.063864 (+ 299us) client_negotiation.cc:240] Negotiation successful
0924 19:05:37.064405 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":308,"threads_started":1}
I20260924 19:05:37.067612 22904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34697 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:05:37 dist-test-slave-9fn1 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 19:05:37.083665 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.068530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57606 (local address 127.0.97.61:34697)
0924 19:05:37.069098 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:37.069104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:37.069127 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:05:37.069179 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:37.069184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:37.069350 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:37.069467 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:37.070436 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.072134 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.073242 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.074226 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.074367 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:37.077810 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:37.077821 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:37.077834 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:37.077861 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:37.080725 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:37.081610 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:37.081617 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:37.081619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:37.081733 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:37.082158 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:37.082161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:37.082163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:37.082475 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:37.082634 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:37.083077 (+ 443us) server_negotiation.cc:299] Negotiation successful
0924 19:05:37.083362 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":428,"thread_start_us":296,"threads_started":1}
I20260924 19:05:37.083777 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.068321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34697 (local address 127.0.0.1:57606)
0924 19:05:37.068539 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:37.068559 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:37.068660 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:37.069530 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:37.069538 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:37.070032 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:05:37.070252 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:37.070265 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.072294 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.072298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:37.073107 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:37.073117 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.074371 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.074375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:37.074518 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:37.075121 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:05:37.075137 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:05:37.077607 (+ 2470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:05:37.080866 (+ 3259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:37.080874 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:37.080881 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:37.081444 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:37.081862 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:37.081870 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:37.081876 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:37.082043 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:37.082584 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:05:37.082592 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:05:37.082766 (+ 174us) client_negotiation.cc:769] Sending connection context
0924 19:05:37.082918 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 19:05:37.083255 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":6}
I20260924 19:05:37.086329 22904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:05:37.087486 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.087106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39823 (local address 127.0.0.1:41030)
0924 19:05:37.087260 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:37.087359 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 19:05:37.087778 22854 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39823: connect: Connection refused (error 111)
I20260924 19:05:37.089346 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.089035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39823 (local address 127.0.0.1:41038)
0924 19:05:37.089168 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:37.089244 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 19:05:37.089643 22904 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39823: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39823: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260924 19:05:37.096232 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39823
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42153,127.0.97.61:34697,127.0.97.60:39823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:05:37.139335 22904 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } attempt: 1
I20260924 19:05:37.140793 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.140175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39823 (local address 127.0.0.1:41042)
0924 19:05:37.140436 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:37.140600 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 19:05:37.141888 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.141585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39823 (local address 127.0.0.1:41050)
0924 19:05:37.141726 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:37.141791 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 19:05:37.142148 22904 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39823: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39823: connect: Connection refused (error 111)
I20260924 19:05:37.210364 22904 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } attempt: 2
I20260924 19:05:37.211598 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.211039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39823 (local address 127.0.0.1:41054)
0924 19:05:37.211285 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:37.211409 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 19:05:37.212638 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.212271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39823 (local address 127.0.0.1:41058)
0924 19:05:37.212417 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:37.212499 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 19:05:37.212855 22904 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39823: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39823: connect: Connection refused (error 111)
I20260924 19:05:37.260887 22830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34697 } attempt: 5
Sep 24 19:05:37 dist-test-slave-9fn1 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 19:05:37.274926 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.261698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34697 (local address 127.0.0.1:57620)
0924 19:05:37.261925 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:37.261946 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:37.262155 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:37.262523 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:37.262528 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:37.262986 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:05:37.263162 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:37.263171 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.264196 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.264202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:37.265050 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:37.265060 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.265868 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.265874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:37.266060 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:37.266759 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:05:37.266776 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:05:37.269022 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:05:37.272679 (+ 3657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:37.272684 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:37.272686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:37.273065 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:37.273451 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:37.273454 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:37.273457 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:37.273653 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:37.274238 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:05:37.274241 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:05:37.274361 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 19:05:37.274493 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 19:05:37.274684 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 19:05:37.275394 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.261790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57620 (local address 127.0.97.61:34697)
0924 19:05:37.261988 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:37.261994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:37.262052 (+ 58us) server_negotiation.cc:407] Connection header received
0924 19:05:37.262303 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:37.262307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:37.262384 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:37.262507 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:37.263318 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.264053 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.265207 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.265709 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.265842 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:37.269230 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:37.269237 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:37.269243 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:37.269281 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:37.272545 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:37.273181 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:37.273186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:37.273190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:37.273342 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:37.273754 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:37.273758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:37.273762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:37.274126 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:37.274279 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:37.274644 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 19:05:37.274908 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 19:05:37.276506 22830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:05:37.277595 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.277152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39823 (local address 127.0.0.1:41074)
0924 19:05:37.277308 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:37.277389 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 19:05:37.278612 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.278289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39823 (local address 127.0.0.1:41086)
0924 19:05:37.278427 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:37.278486 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 19:05:37.278862 22830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39823: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39823: connect: Connection refused (error 111)
I20260924 19:05:37.304090 22904 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } attempt: 3
I20260924 19:05:37.305305 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.304755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39823 (local address 127.0.0.1:41094)
0924 19:05:37.304987 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:37.305096 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 19:05:37.306512 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.306047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39823 (local address 127.0.0.1:41100)
0924 19:05:37.306284 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:37.306380 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260924 19:05:37.306727 22904 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39823: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39823: connect: Connection refused (error 111)
I20260924 19:05:37.330080 22830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } attempt: 1
I20260924 19:05:37.331198 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.330746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39823 (local address 127.0.0.1:41112)
0924 19:05:37.330951 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:37.331038 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 19:05:37.332136 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.331814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39823 (local address 127.0.0.1:41126)
0924 19:05:37.331966 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:37.332035 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 19:05:37.332341 22830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39823: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39823: connect: Connection refused (error 111)
I20260924 19:05:37.400530 22830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } attempt: 2
I20260924 19:05:37.401688 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.401180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39823 (local address 127.0.0.1:41130)
0924 19:05:37.401373 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:37.401473 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 19:05:37.402588 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.402267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39823 (local address 127.0.0.1:41132)
0924 19:05:37.402404 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:37.402474 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 19:05:37.402781 22830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39823: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39823: connect: Connection refused (error 111)
W20260924 19:05:37.422520 22911 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:05:37.422861 22911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:37.422984 22911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:37.423122 22911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:37.430169 22911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:05:37.430286 22911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:37.430413 22911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:37.430496 22911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:05:37.430603 22911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:05:37.440104 22911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42153,127.0.97.61:34697,127.0.97.60:39823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:39823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:37.441340 22911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:37.442593 22911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:05:37.449949 22904 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } attempt: 4
I20260924 19:05:37.451050 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.450528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39823 (local address 127.0.0.1:41134)
0924 19:05:37.450805 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:37.450893 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 19:05:37.451997 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.451661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39823 (local address 127.0.0.1:41140)
0924 19:05:37.451812 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:37.451897 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 19:05:37.452220 22904 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39823: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39823: connect: Connection refused (error 111)
W20260924 19:05:37.455022 22919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:37.455142 22916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:37.455153 22917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:37.456115 22911 server_base.cc:1034] running on GCE node
I20260924 19:05:37.456868 22911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:37.458314 22911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:37.459549 22911 hybrid_clock.cc:648] HybridClock initialized: now 1790276737459489 us; error 36 us; skew 500 ppm
I20260924 19:05:37.459918 22911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:37 dist-test-slave-9fn1 krb5kdc[22750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:37.462997 22911 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260924 19:05:37.464367 22911 webserver.cc:460] Webserver started at http://127.0.97.60:34721/ using document root <none> and password file <none>
I20260924 19:05:37.465044 22911 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:37.465173 22911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:37.465487 22911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:37.468328 22911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-2/data/instance:
uuid: "21b1c12595a84f6397c047451a47f910"
format_stamp: "Formatted at 2026-09-24 19:05:37 on dist-test-slave-9fn1"
server_key: "4ab216e6560f39893f7bc8f221492d02"
server_key_iv: "5af0c39333e3edbd197e88c680ca1b32"
server_key_version: "encryptionkey@0"
I20260924 19:05:37.469081 22911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-2/wal/instance:
uuid: "21b1c12595a84f6397c047451a47f910"
format_stamp: "Formatted at 2026-09-24 19:05:37 on dist-test-slave-9fn1"
server_key: "4ab216e6560f39893f7bc8f221492d02"
server_key_iv: "5af0c39333e3edbd197e88c680ca1b32"
server_key_version: "encryptionkey@0"
I20260924 19:05:37.473938 22911 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 19:05:37.476971 22926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:37.478338 22911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:05:37.478518 22911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-2/wal
uuid: "21b1c12595a84f6397c047451a47f910"
format_stamp: "Formatted at 2026-09-24 19:05:37 on dist-test-slave-9fn1"
server_key: "4ab216e6560f39893f7bc8f221492d02"
server_key_iv: "5af0c39333e3edbd197e88c680ca1b32"
server_key_version: "encryptionkey@0"
I20260924 19:05:37.478718 22911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:37.500622 22911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:37.504673 22911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:37.504959 22911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:37.509001 22830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } attempt: 3
I20260924 19:05:37.510264 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.509722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39823 (local address 127.0.0.1:41156)
0924 19:05:37.509978 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:37.510081 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 19:05:37.511472 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.511024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39823 (local address 127.0.0.1:41168)
0924 19:05:37.511276 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:37.511370 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260924 19:05:37.511705 22830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:39823: Network error: Client connection negotiation failed: client connection to 127.0.97.60:39823: connect: Connection refused (error 111)
I20260924 19:05:37.524123 22911 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:39823
I20260924 19:05:37.524153 22977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:39823 every 8 connection(s)
I20260924 19:05:37.525393 22911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-2/data/info.pb
I20260924 19:05:37.528956 22978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:05:37.529245 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 22911
I20260924 19:05:37.529433 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276386554690-388-0/minicluster-data/master-2/wal/instance
I20260924 19:05:37.529876 388 external_mini_cluster.cc:1428] Setting key 60983ccc7c2513a31551e2d80b630728
I20260924 19:05:37.537794 22978 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42153 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:05:37 dist-test-slave-9fn1 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 24 19:05:37 dist-test-slave-9fn1 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:05:37.553323 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.531546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41176 (local address 127.0.97.60:39823)
0924 19:05:37.532371 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:37.532381 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:37.532417 (+ 36us) server_negotiation.cc:407] Connection header received
0924 19:05:37.534640 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:37.534662 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:37.535067 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:37.535387 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:37.537125 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.538472 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.539904 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.540928 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.541125 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:37.545394 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:37.545403 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:37.545420 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:37.545452 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:37.548867 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:37.549766 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:37.549774 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:37.549782 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:37.549966 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:37.550349 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:37.550355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:37.550358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:37.550843 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:37.551149 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:37.551581 (+ 432us) server_negotiation.cc:299] Negotiation successful
0924 19:05:37.551979 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":405,"threads_started":1}
I20260924 19:05:37.555594 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.540097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59114 (local address 127.0.97.62:42153)
0924 19:05:37.540299 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:37.540305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:37.541054 (+ 749us) server_negotiation.cc:407] Connection header received
0924 19:05:37.541319 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:37.541324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:37.541407 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:37.541555 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:37.542812 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.543866 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.545368 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.545826 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.545918 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:37.549762 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:37.549768 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:37.549772 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:37.549810 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:37.552791 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:37.553461 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:37.553469 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:37.553473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:37.553672 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:37.554043 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:37.554050 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:37.554053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:37.554319 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:37.554488 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:37.554996 (+ 508us) server_negotiation.cc:299] Negotiation successful
0924 19:05:37.555231 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:05:37.556535 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.540154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42153 (local address 127.0.0.1:59114)
0924 19:05:37.540928 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:37.540974 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:37.541172 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:37.541877 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:37.541881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:37.542420 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:05:37.542655 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:37.542663 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.544013 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.544021 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:37.545190 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:37.545205 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.546001 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.546006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:37.546147 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:37.546750 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:05:37.546771 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:05:37.549559 (+ 2788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:05:37.552915 (+ 3356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:37.552927 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:37.552934 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:37.553339 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:37.553771 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:37.553777 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:37.553781 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:37.553929 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:37.554460 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:05:37.554468 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:05:37.554648 (+ 180us) client_negotiation.cc:769] Sending connection context
0924 19:05:37.555874 (+ 1226us) client_negotiation.cc:240] Negotiation successful
0924 19:05:37.556172 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":379,"threads_started":1}
I20260924 19:05:37.557924 22978 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34697 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:05:37 dist-test-slave-9fn1 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 24 19:05:37 dist-test-slave-9fn1 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:05:37.572518 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.558545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34697 (local address 127.0.0.1:57622)
0924 19:05:37.558722 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:37.559065 (+ 343us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:37.559162 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:37.559477 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:37.559482 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:37.559920 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:05:37.560084 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:37.560094 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.561485 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.561490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:37.562496 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:37.562510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.563367 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.563375 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:37.563610 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:37.564116 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:05:37.564139 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:05:37.566741 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:05:37.569978 (+ 3237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:37.569984 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:37.569989 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:37.570465 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:37.570860 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:37.570864 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:37.570868 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:37.571030 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:37.571612 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:05:37.571617 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:05:37.571777 (+ 160us) client_negotiation.cc:769] Sending connection context
0924 19:05:37.571925 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 19:05:37.572205 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 19:05:37.572671 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.558963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57622 (local address 127.0.97.61:34697)
0924 19:05:37.559168 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:37.559173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:37.559192 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:05:37.559265 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:37.559270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:37.559341 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:37.559449 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:37.560249 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.561322 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.562622 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.563204 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.563332 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:37.566900 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:37.566905 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:37.566908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:37.566938 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:37.569854 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:37.570579 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:37.570586 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:37.570592 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:37.570748 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:37.571140 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:37.571145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:37.571149 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:37.571500 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:37.571607 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:37.572035 (+ 428us) server_negotiation.cc:299] Negotiation successful
0924 19:05:37.572300 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:05:37.572816 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.557230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59124 (local address 127.0.97.62:42153)
0924 19:05:37.557475 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:37.557478 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:37.557491 (+ 13us) server_negotiation.cc:407] Connection header received
0924 19:05:37.557582 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:37.557585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:37.557645 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:37.558348 (+ 703us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:37.559269 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.560389 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.561616 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.562308 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.562425 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:37.567522 (+ 5097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:37.567526 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:37.567530 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:37.567557 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:37.570177 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:37.570860 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:37.570866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:37.570869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:37.571007 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:37.571406 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:37.571409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:37.571411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:37.571730 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:37.572007 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:37.572299 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 19:05:37.572494 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260924 19:05:37.573714 22978 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:05:37 dist-test-slave-9fn1 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 24 19:05:37 dist-test-slave-9fn1 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 19:05:37.586642 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.574575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39823 (local address 127.0.0.1:41184)
0924 19:05:37.574802 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:37.574817 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:37.574913 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:37.575156 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:37.575160 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:37.575503 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:05:37.575635 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:37.575643 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.576541 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.576548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:37.577344 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:37.577354 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.578075 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.578080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:37.578274 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:37.578803 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:05:37.578817 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:05:37.580816 (+ 1999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:05:37.584343 (+ 3527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:37.584347 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:37.584349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:37.584798 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:37.585182 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:37.585185 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:37.585188 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:37.585303 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:37.585889 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:05:37.585892 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:05:37.585996 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 19:05:37.586099 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 19:05:37.586299 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260924 19:05:37.586733 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.574627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41184 (local address 127.0.97.60:39823)
0924 19:05:37.574771 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:37.574776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:37.574878 (+ 102us) server_negotiation.cc:407] Connection header received
0924 19:05:37.574996 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:37.575001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:37.575053 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:37.575143 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:37.575762 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.576425 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.577470 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.577910 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.578062 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:37.580981 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:37.580986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:37.580989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:37.581014 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:37.584209 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:37.584919 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:37.584927 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:37.584934 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:37.585077 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:37.585420 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:37.585425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:37.585430 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:37.585778 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:37.585927 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:37.586220 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 19:05:37.586457 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 19:05:37.589124 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.574562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57626 (local address 127.0.97.61:34697)
0924 19:05:37.574729 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:37.574735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:37.574923 (+ 188us) server_negotiation.cc:407] Connection header received
0924 19:05:37.575088 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:37.575093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:37.575162 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:37.575252 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:37.576340 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.576968 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.578174 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.578583 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.578704 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:37.583056 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:37.583062 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:37.583066 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:37.583104 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:37.586347 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:37.587137 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:37.587142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:37.587146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:37.587290 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:37.587674 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:37.587680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:37.587683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:37.588045 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:37.588324 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:37.588591 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 19:05:37.588806 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 19:05:37.592244 22978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21b1c12595a84f6397c047451a47f910: Bootstrap starting.
I20260924 19:05:37.593688 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:05:37.595366 22978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21b1c12595a84f6397c047451a47f910: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:37.596418 22978 log.cc:824] T 00000000000000000000000000000000 P 21b1c12595a84f6397c047451a47f910: Log is configured to *not* fsync() on all Append() calls
I20260924 19:05:37.596150 22805 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "53ce32c546cb494cbe7111c9358a7c20"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59124
I20260924 19:05:37.599071 22978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21b1c12595a84f6397c047451a47f910: No bootstrap required, opened a new log
I20260924 19:05:37.605883 22978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21b1c12595a84f6397c047451a47f910 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ce32c546cb494cbe7111c9358a7c20" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42153 } } peers { permanent_uuid: "6d5fa1a56dc049f78281d8904f98af0a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34697 } } peers { permanent_uuid: "21b1c12595a84f6397c047451a47f910" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } }
I20260924 19:05:37.606196 22978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21b1c12595a84f6397c047451a47f910 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b1c12595a84f6397c047451a47f910, State: Initialized, Role: FOLLOWER
I20260924 19:05:37.606642 22978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21b1c12595a84f6397c047451a47f910 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ce32c546cb494cbe7111c9358a7c20" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42153 } } peers { permanent_uuid: "6d5fa1a56dc049f78281d8904f98af0a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34697 } } peers { permanent_uuid: "21b1c12595a84f6397c047451a47f910" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } }
I20260924 19:05:37.607580 22984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21b1c12595a84f6397c047451a47f910 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ce32c546cb494cbe7111c9358a7c20" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42153 } } peers { permanent_uuid: "6d5fa1a56dc049f78281d8904f98af0a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34697 } } peers { permanent_uuid: "21b1c12595a84f6397c047451a47f910" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } } }
I20260924 19:05:37.607969 22984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21b1c12595a84f6397c047451a47f910 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:05:37.608546 22978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21b1c12595a84f6397c047451a47f910 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 19:05:37.617295 22995 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 21b1c12595a84f6397c047451a47f910: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:05:37.617455 22995 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:05:37.615473 22911 master_runner.cc:186] Error getting master registration for 127.0.97.62:42153: OK, instance_id { permanent_uuid: "53ce32c546cb494cbe7111c9358a7c20" instance_seqno: 1790276736547440 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:05:37.626842 22911 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:05:37.688956 22830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } attempt: 4
Sep 24 19:05:37 dist-test-slave-9fn1 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260924 19:05:37.704577 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.689799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39823 (local address 127.0.0.1:41192)
0924 19:05:37.690018 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:37.690039 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:37.690223 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:37.690652 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:37.690658 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:37.691149 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:05:37.691366 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:37.691378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.692819 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.692825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:37.694343 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:37.694357 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.695362 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.695368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:37.695571 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:37.696254 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:05:37.696277 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:05:37.698958 (+ 2681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:05:37.702184 (+ 3226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:37.702193 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:37.702196 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:37.702549 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:37.702918 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:37.702924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:37.702928 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:37.703068 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:37.703588 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:05:37.703593 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:05:37.703741 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 19:05:37.703895 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 19:05:37.704186 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 19:05:37.704581 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.689853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41192 (local address 127.0.97.60:39823)
0924 19:05:37.690034 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:37.690041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:37.690108 (+ 67us) server_negotiation.cc:407] Connection header received
0924 19:05:37.690384 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:37.690391 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:37.690475 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:37.690588 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:37.691524 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.692667 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.694484 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.695205 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.695329 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:37.699189 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:37.699193 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:37.699196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:37.699224 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:37.702056 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:37.702664 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:37.702671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:37.702675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:37.702805 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:37.703180 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:37.703185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:37.703189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:37.703457 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:37.703609 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:37.704011 (+ 402us) server_negotiation.cc:299] Negotiation successful
0924 19:05:37.704223 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 19:05:37.710397 22830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53ce32c546cb494cbe7111c9358a7c20: Bootstrap starting.
I20260924 19:05:37.713802 22830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53ce32c546cb494cbe7111c9358a7c20: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:37.714771 22830 log.cc:824] T 00000000000000000000000000000000 P 53ce32c546cb494cbe7111c9358a7c20: Log is configured to *not* fsync() on all Append() calls
I20260924 19:05:37.717263 22830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53ce32c546cb494cbe7111c9358a7c20: No bootstrap required, opened a new log
I20260924 19:05:37.722929 22830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53ce32c546cb494cbe7111c9358a7c20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ce32c546cb494cbe7111c9358a7c20" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42153 } } peers { permanent_uuid: "6d5fa1a56dc049f78281d8904f98af0a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34697 } } peers { permanent_uuid: "21b1c12595a84f6397c047451a47f910" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } }
I20260924 19:05:37.723302 22830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53ce32c546cb494cbe7111c9358a7c20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53ce32c546cb494cbe7111c9358a7c20, State: Initialized, Role: FOLLOWER
I20260924 19:05:37.723771 22830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53ce32c546cb494cbe7111c9358a7c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ce32c546cb494cbe7111c9358a7c20" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42153 } } peers { permanent_uuid: "6d5fa1a56dc049f78281d8904f98af0a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34697 } } peers { permanent_uuid: "21b1c12595a84f6397c047451a47f910" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } }
I20260924 19:05:37.724728 22998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53ce32c546cb494cbe7111c9358a7c20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ce32c546cb494cbe7111c9358a7c20" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42153 } } peers { permanent_uuid: "6d5fa1a56dc049f78281d8904f98af0a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34697 } } peers { permanent_uuid: "21b1c12595a84f6397c047451a47f910" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } } }
I20260924 19:05:37.725073 22998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53ce32c546cb494cbe7111c9358a7c20 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:05:37.725838 22830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53ce32c546cb494cbe7111c9358a7c20 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:05:37.725927 22805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53ce32c546cb494cbe7111c9358a7c20 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 19:05:37.726131 22805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53ce32c546cb494cbe7111c9358a7c20 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:37.729568 22805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53ce32c546cb494cbe7111c9358a7c20 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ce32c546cb494cbe7111c9358a7c20" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42153 } } peers { permanent_uuid: "6d5fa1a56dc049f78281d8904f98af0a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34697 } } peers { permanent_uuid: "21b1c12595a84f6397c047451a47f910" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } }
I20260924 19:05:37.731950 22953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53ce32c546cb494cbe7111c9358a7c20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "21b1c12595a84f6397c047451a47f910"
I20260924 19:05:37.732438 22953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21b1c12595a84f6397c047451a47f910 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:37.735833 22953 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 21b1c12595a84f6397c047451a47f910 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53ce32c546cb494cbe7111c9358a7c20 in term 1.
I20260924 19:05:37.736469 22780 leader_election.cc:302] T 00000000000000000000000000000000 P 53ce32c546cb494cbe7111c9358a7c20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21b1c12595a84f6397c047451a47f910, 53ce32c546cb494cbe7111c9358a7c20; no voters:
I20260924 19:05:37.736966 22805 leader_election.cc:288] T 00000000000000000000000000000000 P 53ce32c546cb494cbe7111c9358a7c20 [CANDIDATE]: Term 1 election: Requested vote from peers 6d5fa1a56dc049f78281d8904f98af0a (127.0.97.61:34697), 21b1c12595a84f6397c047451a47f910 (127.0.97.60:39823)
I20260924 19:05:37.736966 22998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53ce32c546cb494cbe7111c9358a7c20 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:05:37.751318 22998 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53ce32c546cb494cbe7111c9358a7c20 [term 1 LEADER]: Becoming Leader. State: Replica: 53ce32c546cb494cbe7111c9358a7c20, State: Running, Role: LEADER
I20260924 19:05:37.751394 22904 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } attempt: 5
I20260924 19:05:37.751891 22998 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53ce32c546cb494cbe7111c9358a7c20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ce32c546cb494cbe7111c9358a7c20" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42153 } } peers { permanent_uuid: "6d5fa1a56dc049f78281d8904f98af0a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34697 } } peers { permanent_uuid: "21b1c12595a84f6397c047451a47f910" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } }
W20260924 19:05:37.757040 22932 tablet.cc:2369] T 00000000000000000000000000000000 P 21b1c12595a84f6397c047451a47f910: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:05:37.758797 23006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53ce32c546cb494cbe7111c9358a7c20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53ce32c546cb494cbe7111c9358a7c20. Latest consensus state: current_term: 1 leader_uuid: "53ce32c546cb494cbe7111c9358a7c20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ce32c546cb494cbe7111c9358a7c20" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42153 } } peers { permanent_uuid: "6d5fa1a56dc049f78281d8904f98af0a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34697 } } peers { permanent_uuid: "21b1c12595a84f6397c047451a47f910" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } } }
I20260924 19:05:37.759222 23006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53ce32c546cb494cbe7111c9358a7c20 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:37.763162 23012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 24 19:05:37 dist-test-slave-9fn1 krb5kdc[22750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260924 19:05:37.776633 23012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:05:37.779734 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.752854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41206 (local address 127.0.97.60:39823)
0924 19:05:37.753036 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:37.753042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:37.753061 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:05:37.753122 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:37.753130 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:37.753231 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:37.753360 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:37.754796 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.756243 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.758320 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.758970 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.759093 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:37.766270 (+ 7177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:37.766280 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:37.766287 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:37.766336 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:37.772383 (+ 6047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:37.773924 (+ 1541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:37.773960 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:37.773965 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:37.774248 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:37.775018 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:37.775066 (+ 48us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:37.775072 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:37.775799 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:37.776329 (+ 530us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:37.776943 (+ 614us) server_negotiation.cc:299] Negotiation successful
0924 19:05:37.777959 (+ 1016us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 19:05:37.779930 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.752079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:39823 (local address 127.0.0.1:41206)
0924 19:05:37.752460 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:37.752484 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:37.752594 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:37.753583 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:37.753592 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:37.754270 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:05:37.754535 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:37.754549 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.756475 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.756481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:37.758093 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:37.758112 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.759354 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.759387 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:37.759813 (+ 426us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:37.760900 (+ 1087us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:05:37.760958 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:05:37.765947 (+ 4989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:05:37.772613 (+ 6666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:37.772643 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:37.772651 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:37.773703 (+ 1052us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:37.774501 (+ 798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:37.774508 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:37.774514 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:37.774823 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:37.775998 (+ 1175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:05:37.776007 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:05:37.776487 (+ 480us) client_negotiation.cc:769] Sending connection context
0924 19:05:37.777137 (+ 650us) client_negotiation.cc:240] Negotiation successful
0924 19:05:37.778378 (+ 1241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243}
W20260924 19:05:37.786834 22784 tablet.cc:2369] T 00000000000000000000000000000000 P 53ce32c546cb494cbe7111c9358a7c20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:05:37.787189 22904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d5fa1a56dc049f78281d8904f98af0a: Bootstrap starting.
I20260924 19:05:37.793519 22904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d5fa1a56dc049f78281d8904f98af0a: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:37.796342 22953 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 21b1c12595a84f6397c047451a47f910 [term 1 FOLLOWER]: Refusing update from remote peer 53ce32c546cb494cbe7111c9358a7c20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:05:37.797384 23006 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 53ce32c546cb494cbe7111c9358a7c20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b1c12595a84f6397c047451a47f910" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:05:37.802232 22904 log.cc:824] T 00000000000000000000000000000000 P 6d5fa1a56dc049f78281d8904f98af0a: Log is configured to *not* fsync() on all Append() calls
I20260924 19:05:37.821406 22984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21b1c12595a84f6397c047451a47f910 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53ce32c546cb494cbe7111c9358a7c20. Latest consensus state: current_term: 1 leader_uuid: "53ce32c546cb494cbe7111c9358a7c20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ce32c546cb494cbe7111c9358a7c20" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42153 } } peers { permanent_uuid: "6d5fa1a56dc049f78281d8904f98af0a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34697 } } peers { permanent_uuid: "21b1c12595a84f6397c047451a47f910" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } } }
I20260924 19:05:37.821823 22984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21b1c12595a84f6397c047451a47f910 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:05:37.823879 23017 mvcc.cc:205] Tried to move back new op lower bound from 7332973518000463872 to 7332973517837074432. Current Snapshot: MvccSnapshot[applied={T|T < 7332973518000463872}]
I20260924 19:05:37.824859 22984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21b1c12595a84f6397c047451a47f910 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "53ce32c546cb494cbe7111c9358a7c20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ce32c546cb494cbe7111c9358a7c20" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42153 } } peers { permanent_uuid: "6d5fa1a56dc049f78281d8904f98af0a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34697 } } peers { permanent_uuid: "21b1c12595a84f6397c047451a47f910" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } } }
I20260924 19:05:37.825166 22984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21b1c12595a84f6397c047451a47f910 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:05:37.837996 23006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53ce32c546cb494cbe7111c9358a7c20 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "53ce32c546cb494cbe7111c9358a7c20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ce32c546cb494cbe7111c9358a7c20" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42153 } } peers { permanent_uuid: "6d5fa1a56dc049f78281d8904f98af0a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34697 } } peers { permanent_uuid: "21b1c12595a84f6397c047451a47f910" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } } }
I20260924 19:05:37.838311 23006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53ce32c546cb494cbe7111c9358a7c20 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:37.840279 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.793265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59136 (local address 127.0.97.62:42153)
0924 19:05:37.793463 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:37.793468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:37.800597 (+ 7129us) server_negotiation.cc:407] Connection header received
0924 19:05:37.803510 (+ 2913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:37.803519 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:37.803621 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:37.803698 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:37.811591 (+ 7893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.812970 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.825879 (+ 12909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.826886 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.827013 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:37.831480 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:37.831487 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:37.831493 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:37.831539 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:37.835458 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:37.836526 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:37.836535 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:37.836541 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:37.836758 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:37.837292 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:37.837314 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:37.837320 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:37.837871 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:37.838436 (+ 565us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:37.839222 (+ 786us) server_negotiation.cc:299] Negotiation successful
0924 19:05:37.839735 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:05:37.840152 22904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d5fa1a56dc049f78281d8904f98af0a: No bootstrap required, opened a new log
I20260924 19:05:37.843287 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.770326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41220 (local address 127.0.97.60:39823)
0924 19:05:37.793453 (+ 23127us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:37.793464 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:37.800638 (+ 7174us) server_negotiation.cc:407] Connection header received
0924 19:05:37.800704 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:37.800709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:37.800790 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:37.800892 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:37.806086 (+ 5194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.807265 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.815407 (+ 8142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.816135 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.816272 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:37.819743 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:37.819750 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:37.819755 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:37.819792 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:37.830819 (+ 11027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:37.832210 (+ 1391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:37.832250 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:37.832255 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:37.832541 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:37.833413 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:37.833418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:37.833422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:37.837216 (+ 3794us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:37.840862 (+ 3646us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:37.842567 (+ 1705us) server_negotiation.cc:299] Negotiation successful
0924 19:05:37.842864 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22527,"thread_start_us":28576,"threads_started":1}
I20260924 19:05:37.849769 23019 mvcc.cc:205] Tried to move back new op lower bound from 7332973518000463872 to 7332973517837074432. Current Snapshot: MvccSnapshot[applied={T|T < 7332973518000463872}]
I20260924 19:05:37.849965 23012 catalog_manager.cc:1347] Generated new cluster ID: ab2095f97a2f4a7494ecb10c242ee5f1
I20260924 19:05:37.850052 23012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:05:37.858013 22904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d5fa1a56dc049f78281d8904f98af0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ce32c546cb494cbe7111c9358a7c20" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42153 } } peers { permanent_uuid: "6d5fa1a56dc049f78281d8904f98af0a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34697 } } peers { permanent_uuid: "21b1c12595a84f6397c047451a47f910" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } }
I20260924 19:05:37.858423 22904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d5fa1a56dc049f78281d8904f98af0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d5fa1a56dc049f78281d8904f98af0a, State: Initialized, Role: FOLLOWER
I20260924 19:05:37.858973 22904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d5fa1a56dc049f78281d8904f98af0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ce32c546cb494cbe7111c9358a7c20" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42153 } } peers { permanent_uuid: "6d5fa1a56dc049f78281d8904f98af0a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34697 } } peers { permanent_uuid: "21b1c12595a84f6397c047451a47f910" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } }
I20260924 19:05:37.866691 23028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d5fa1a56dc049f78281d8904f98af0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ce32c546cb494cbe7111c9358a7c20" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42153 } } peers { permanent_uuid: "6d5fa1a56dc049f78281d8904f98af0a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34697 } } peers { permanent_uuid: "21b1c12595a84f6397c047451a47f910" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } } }
I20260924 19:05:37.867069 23028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d5fa1a56dc049f78281d8904f98af0a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:05:37.868865 22904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d5fa1a56dc049f78281d8904f98af0a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:05:37.868965 22879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "53ce32c546cb494cbe7111c9358a7c20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6d5fa1a56dc049f78281d8904f98af0a"
I20260924 19:05:37.869438 22879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d5fa1a56dc049f78281d8904f98af0a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:37.872855 22879 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6d5fa1a56dc049f78281d8904f98af0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53ce32c546cb494cbe7111c9358a7c20 in term 1.
I20260924 19:05:37.874418 22878 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6d5fa1a56dc049f78281d8904f98af0a [term 1 FOLLOWER]: Refusing update from remote peer 53ce32c546cb494cbe7111c9358a7c20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:05:37.875319 23006 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 53ce32c546cb494cbe7111c9358a7c20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d5fa1a56dc049f78281d8904f98af0a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:05:37.878779 23027 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 19:05:37.881017 23012 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:05:37.886318 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:37.808700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57636 (local address 127.0.97.61:34697)
0924 19:05:37.808898 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:37.808904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:37.819720 (+ 10816us) server_negotiation.cc:407] Connection header received
0924 19:05:37.824666 (+ 4946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:37.824675 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:37.824784 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:37.824877 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:37.845638 (+ 20761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.846687 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.849653 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:37.850398 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:37.850549 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:37.857178 (+ 6629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:37.857187 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:37.857193 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:37.857239 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:37.861219 (+ 3980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:37.862760 (+ 1541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:37.862766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:37.862770 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:37.862928 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:37.863720 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:37.863726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:37.863729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:37.864103 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:37.866119 (+ 2016us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:37.885703 (+ 19584us) server_negotiation.cc:299] Negotiation successful
0924 19:05:37.885964 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 19:05:37.887578 23012 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:05:37.882541 22763 master_runner.cc:186] Error getting master registration for 127.0.97.61:34697: OK, instance_id { permanent_uuid: "6d5fa1a56dc049f78281d8904f98af0a" instance_seqno: 1790276737027668 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:05:37.905531 22763 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:05:37.912664 23012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53ce32c546cb494cbe7111c9358a7c20: Generated new TSK 0
I20260924 19:05:37.913678 23012 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:05:37.951653 22998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53ce32c546cb494cbe7111c9358a7c20 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "53ce32c546cb494cbe7111c9358a7c20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ce32c546cb494cbe7111c9358a7c20" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42153 } } peers { permanent_uuid: "6d5fa1a56dc049f78281d8904f98af0a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34697 } } peers { permanent_uuid: "21b1c12595a84f6397c047451a47f910" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } } }
I20260924 19:05:37.951642 23028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d5fa1a56dc049f78281d8904f98af0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53ce32c546cb494cbe7111c9358a7c20. Latest consensus state: current_term: 1 leader_uuid: "53ce32c546cb494cbe7111c9358a7c20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ce32c546cb494cbe7111c9358a7c20" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42153 } } peers { permanent_uuid: "6d5fa1a56dc049f78281d8904f98af0a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34697 } } peers { permanent_uuid: "21b1c12595a84f6397c047451a47f910" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } } }
I20260924 19:05:37.951998 23028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d5fa1a56dc049f78281d8904f98af0a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:05:37.952023 22998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53ce32c546cb494cbe7111c9358a7c20 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:37.951808 23033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d5fa1a56dc049f78281d8904f98af0a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "53ce32c546cb494cbe7111c9358a7c20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53ce32c546cb494cbe7111c9358a7c20" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42153 } } peers { permanent_uuid: "6d5fa1a56dc049f78281d8904f98af0a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34697 } } peers { permanent_uuid: "21b1c12595a84f6397c047451a47f910" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 39823 } } }
I20260924 19:05:37.952158 23033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d5fa1a56dc049f78281d8904f98af0a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:05:37.990222 23046 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260924 19:05:38.020699 22981 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:05:38.008885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41224 (local address 127.0.97.60:39823)
0924 19:05:38.009056 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:38.009061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:38.018118 (+ 9057us) server_negotiation.cc:407] Connection header received
0924 19:05:38.018487 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:38.018497 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:38.018603 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:38.018716 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:38.020312 (+ 1596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41224: BlockingRecv error: recv got EOF from 127.0.0.1:41224 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 19:05:38.023294 22833 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:05:38.007843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59148 (local address 127.0.97.62:42153)
0924 19:05:38.008704 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:38.008714 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:38.021605 (+ 12891us) server_negotiation.cc:407] Connection header received
0924 19:05:38.021937 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:38.021943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:38.022058 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:38.022181 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:38.023066 (+ 885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59148: BlockingRecv error: recv got EOF from 127.0.0.1:59148 (error 108)
Metrics: {"server-negotiator.queue_time_us":651}
W20260924 19:05:38.029807 22908 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:05:38.027387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57646 (local address 127.0.97.61:34697)
0924 19:05:38.027587 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:38.027593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:38.028310 (+ 717us) server_negotiation.cc:407] Connection header received
0924 19:05:38.028459 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:38.028464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:38.028556 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:38.028669 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:38.029585 (+ 916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57646: BlockingRecv error: recv got EOF from 127.0.0.1:57646 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 19:05:38.066380 22908 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:05:38.063939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57658 (local address 127.0.97.61:34697)
0924 19:05:38.064127 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:38.064133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:38.064666 (+ 533us) server_negotiation.cc:407] Connection header received
0924 19:05:38.064994 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:38.065000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:38.065146 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:38.065258 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:38.066113 (+ 855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57658: BlockingRecv error: recv got EOF from 127.0.0.1:57658 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 19:05:38.075497 22908 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:05:38.070706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57664 (local address 127.0.97.61:34697)
0924 19:05:38.070857 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:38.070862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:38.074085 (+ 3223us) server_negotiation.cc:407] Connection header received
0924 19:05:38.074272 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:38.074278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:38.074362 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:38.074468 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:38.075289 (+ 821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57664: BlockingRecv error: recv got EOF from 127.0.0.1:57664 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 19:05:38.082616 22908 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:05:38.080526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57666 (local address 127.0.97.61:34697)
0924 19:05:38.080685 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:38.080691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:38.081210 (+ 519us) server_negotiation.cc:407] Connection header received
0924 19:05:38.081395 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:38.081400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:38.081492 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:38.081615 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:38.082382 (+ 767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57666: BlockingRecv error: recv got EOF from 127.0.0.1:57666 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 19:05:38.093664 22908 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:05:38.089714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57668 (local address 127.0.97.61:34697)
0924 19:05:38.089909 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:38.089915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:38.092153 (+ 2238us) server_negotiation.cc:407] Connection header received
0924 19:05:38.092341 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:38.092354 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:38.092456 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:38.092572 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:38.093406 (+ 834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57668: BlockingRecv error: recv got EOF from 127.0.0.1:57668 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260924 19:05:38.108685 22908 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:05:38.106547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57672 (local address 127.0.97.61:34697)
0924 19:05:38.106717 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:38.106722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:38.107262 (+ 540us) server_negotiation.cc:407] Connection header received
0924 19:05:38.107433 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:38.107439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:38.107528 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:38.107638 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:38.108433 (+ 795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57672: BlockingRecv error: recv got EOF from 127.0.0.1:57672 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 19:05:38.130339 22908 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:05:38.128000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57678 (local address 127.0.97.61:34697)
0924 19:05:38.128192 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:38.128198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:38.128716 (+ 518us) server_negotiation.cc:407] Connection header received
0924 19:05:38.128906 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:38.128913 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:38.129008 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:38.129132 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:38.130045 (+ 913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57678: BlockingRecv error: recv got EOF from 127.0.0.1:57678 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 19:05:38.168048 22908 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:05:38.165770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57692 (local address 127.0.97.61:34697)
0924 19:05:38.166038 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:38.166048 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:38.166560 (+ 512us) server_negotiation.cc:407] Connection header received
0924 19:05:38.166776 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:38.166780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:38.166865 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:38.166950 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:38.167780 (+ 830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57692: BlockingRecv error: recv got EOF from 127.0.0.1:57692 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
W20260924 19:05:38.237960 22908 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:05:38.235965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57702 (local address 127.0.97.61:34697)
0924 19:05:38.236215 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:38.236222 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:38.236612 (+ 390us) server_negotiation.cc:407] Connection header received
0924 19:05:38.236784 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:38.236790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:38.236893 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:38.237027 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:38.237745 (+ 718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57702: BlockingRecv error: recv got EOF from 127.0.0.1:57702 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260924 19:05:38.371888 22908 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:05:38.370191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57710 (local address 127.0.97.61:34697)
0924 19:05:38.370408 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:38.370412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:38.370733 (+ 321us) server_negotiation.cc:407] Connection header received
0924 19:05:38.370889 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:38.370894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:38.370964 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:38.371061 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:38.371691 (+ 630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57710: BlockingRecv error: recv got EOF from 127.0.0.1:57710 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 19:05:38.619387 22995 catalog_manager.cc:1259] Loaded cluster ID: ab2095f97a2f4a7494ecb10c242ee5f1
I20260924 19:05:38.619527 22995 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 21b1c12595a84f6397c047451a47f910: loading cluster ID for follower catalog manager: success
I20260924 19:05:38.622401 22995 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 21b1c12595a84f6397c047451a47f910: acquiring CA information for follower catalog manager: success
I20260924 19:05:38.624111 22995 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 21b1c12595a84f6397c047451a47f910: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 19:05:38.632812 22908 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:05:38.631198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57722 (local address 127.0.97.61:34697)
0924 19:05:38.631406 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:38.631410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:38.631670 (+ 260us) server_negotiation.cc:407] Connection header received
0924 19:05:38.631838 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:38.631841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:38.631914 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:38.632011 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:38.632646 (+ 635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57722: BlockingRecv error: recv got EOF from 127.0.0.1:57722 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260924 19:05:38.994679 23046 catalog_manager.cc:1259] Loaded cluster ID: ab2095f97a2f4a7494ecb10c242ee5f1
I20260924 19:05:38.994859 23046 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6d5fa1a56dc049f78281d8904f98af0a: loading cluster ID for follower catalog manager: success
I20260924 19:05:38.997861 23046 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6d5fa1a56dc049f78281d8904f98af0a: acquiring CA information for follower catalog manager: success
I20260924 19:05:38.999919 23046 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6d5fa1a56dc049f78281d8904f98af0a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:05:39.156216 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:39.148494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57732 (local address 127.0.97.61:34697)
0924 19:05:39.149427 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:39.149435 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:39.149462 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:05:39.149566 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:39.149573 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:39.149806 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:39.149987 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:05:39.150476 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:39.151722 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:39.153482 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:39.154198 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:39.154325 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:39.154809 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:05:39.155553 (+ 744us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:05:39.155719 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:39.155783 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 19:05:39.155869 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":776,"thread_start_us":609,"threads_started":1}
I20260924 19:05:39.168648 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:39.162920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41228 (local address 127.0.97.60:39823)
0924 19:05:39.163539 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:39.163546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:39.163566 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:05:39.163668 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:39.163672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:39.163823 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:39.164004 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:05:39.164396 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:39.165087 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:39.166355 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:39.166769 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:39.166908 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:39.167357 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:05:39.167966 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:05:39.168114 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:39.168232 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 19:05:39.168296 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":385,"threads_started":1}
I20260924 19:05:39.189250 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:39.175969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41244 (local address 127.0.97.60:39823)
0924 19:05:39.176166 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:39.176175 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:39.182326 (+ 6151us) server_negotiation.cc:407] Connection header received
0924 19:05:39.182411 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:39.182416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:39.182509 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:39.182604 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:05:39.183376 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:39.184453 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:39.187121 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:39.187709 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:39.187800 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:39.188476 (+ 676us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:05:39.188616 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:05:39.188678 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:39.188904 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 19:05:39.188988 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 19:05:39.189827 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:39.176168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59164 (local address 127.0.97.62:42153)
0924 19:05:39.182620 (+ 6452us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:39.182626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:39.182704 (+ 78us) server_negotiation.cc:407] Connection header received
0924 19:05:39.182820 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:39.182826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:39.183098 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:39.183274 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:05:39.184020 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:39.185072 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:39.187251 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:39.187838 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:39.187958 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:39.188676 (+ 718us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:05:39.189098 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:05:39.189175 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:39.189416 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 19:05:39.189523 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6310,"thread_start_us":6244,"threads_started":1}
I20260924 19:05:39.190737 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:39.182224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57740 (local address 127.0.97.61:34697)
0924 19:05:39.182395 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:39.182401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:39.184047 (+ 1646us) server_negotiation.cc:407] Connection header received
0924 19:05:39.184298 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:39.184303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:39.184395 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:39.184463 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:05:39.185452 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:39.186613 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:39.188644 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:39.189225 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:39.189329 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:39.189942 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:05:39.190062 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:05:39.190128 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:39.190337 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 19:05:39.190433 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 19:05:39.195142 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 22763
I20260924 19:05:39.196367 22825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:05:39.306329 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 22763
I20260924 19:05:39.327629 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 22837
I20260924 19:05:39.328840 22899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:05:39.443557 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 22837
I20260924 19:05:39.459618 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 22911
I20260924 19:05:39.460695 22973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:05:39.567536 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 22911
2026-09-24T19:05:39Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4756 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:05:39 dist-test-slave-9fn1 krb5kdc[23129](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:05:39 dist-test-slave-9fn1 krb5kdc[23129](info): set up 2 sockets
Sep 24 19:05:39 dist-test-slave-9fn1 krb5kdc[23129](info): commencing operation
krb5kdc: starting...
W20260924 19:05:40.666013 388 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/krb5kdc/test-user.keytab.
Sep 24 19:05:40 dist-test-slave-9fn1 krb5kdc[23129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:05:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:05:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260924 19:05:40.914635 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38773
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38773,127.0.97.61:36443,127.0.97.60:45101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:41.238373 23142 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:05:41.238726 23142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:41.238853 23142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:41.239017 23142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:41.246068 23142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:05:41.246143 23142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:41.246266 23142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:41.246352 23142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:05:41.246443 23142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:05:41.255681 23142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38773,127.0.97.61:36443,127.0.97.60:45101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:41.256880 23142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:41.258165 23142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:41.269522 23148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:41.269578 23147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:41.269570 23150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:41.270561 23142 server_base.cc:1034] running on GCE node
I20260924 19:05:41.271220 23142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:41.272475 23142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:41.273689 23142 hybrid_clock.cc:648] HybridClock initialized: now 1790276741273627 us; error 29 us; skew 500 ppm
I20260924 19:05:41.274037 23142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:41 dist-test-slave-9fn1 krb5kdc[23129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:41.281286 23142 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260924 19:05:41.282716 23142 webserver.cc:460] Webserver started at http://127.0.97.62:44593/ using document root <none> and password file <none>
I20260924 19:05:41.283403 23142 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:41.283533 23142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:41.283850 23142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:41.286835 23142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-0/data/instance:
uuid: "c8493804eaad4c1fba9431b9e2c317d0"
format_stamp: "Formatted at 2026-09-24 19:05:41 on dist-test-slave-9fn1"
server_key: "1df39ad4cbf7396c9567688d36144273"
server_key_iv: "dac0d132c55921a57174b1c189db3497"
server_key_version: "encryptionkey@0"
I20260924 19:05:41.287971 23142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-0/wal/instance:
uuid: "c8493804eaad4c1fba9431b9e2c317d0"
format_stamp: "Formatted at 2026-09-24 19:05:41 on dist-test-slave-9fn1"
server_key: "1df39ad4cbf7396c9567688d36144273"
server_key_iv: "dac0d132c55921a57174b1c189db3497"
server_key_version: "encryptionkey@0"
I20260924 19:05:41.292908 23142 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 19:05:41.296360 23157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:41.297783 23142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:05:41.298002 23142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-0/wal
uuid: "c8493804eaad4c1fba9431b9e2c317d0"
format_stamp: "Formatted at 2026-09-24 19:05:41 on dist-test-slave-9fn1"
server_key: "1df39ad4cbf7396c9567688d36144273"
server_key_iv: "dac0d132c55921a57174b1c189db3497"
server_key_version: "encryptionkey@0"
I20260924 19:05:41.298228 23142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:41.309327 23142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:41.313679 23142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:41.313956 23142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:41.332370 23142 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38773
I20260924 19:05:41.332401 23208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38773 every 8 connection(s)
I20260924 19:05:41.333716 23142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-0/data/info.pb
I20260924 19:05:41.336870 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 23142
I20260924 19:05:41.337064 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-0/wal/instance
I20260924 19:05:41.337342 23209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:05:41.337610 388 external_mini_cluster.cc:1428] Setting key 37d9b0fee1dd1346bf4d42a71c3e6859
I20260924 19:05:41.343689 23209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:05:41 dist-test-slave-9fn1 krb5kdc[23129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260924 19:05:41.365737 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:41.346722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38773 (local address 127.0.0.1:48740)
0924 19:05:41.347504 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:41.347549 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:41.349632 (+ 2083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:41.350445 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:41.350451 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:41.351119 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:05:41.351443 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:41.351459 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:41.352942 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:41.352948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:41.354084 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:41.354095 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:41.354969 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:41.354975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:41.355185 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:41.355836 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:05:41.355857 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:05:41.358252 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:05:41.361577 (+ 3325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:41.361586 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:41.361595 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:41.361984 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:41.362444 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:41.362449 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:41.362453 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:41.362601 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:41.363322 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:05:41.363331 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:05:41.363530 (+ 199us) client_negotiation.cc:769] Sending connection context
0924 19:05:41.363792 (+ 262us) client_negotiation.cc:240] Negotiation successful
0924 19:05:41.364397 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":153,"thread_start_us":341,"threads_started":1}
I20260924 19:05:41.365706 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:41.347997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48740 (local address 127.0.97.62:38773)
0924 19:05:41.348767 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:41.348774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:41.348824 (+ 50us) server_negotiation.cc:407] Connection header received
0924 19:05:41.349989 (+ 1165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:41.349994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:41.350237 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:41.350439 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:41.351639 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:41.352793 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:41.354213 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:41.354754 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:41.354951 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:41.358467 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:41.358474 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:41.358485 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:41.358510 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:41.361361 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:41.362125 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:41.362131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:41.362133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:41.362331 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:41.362710 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:41.362715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:41.362718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:41.363187 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:41.363390 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:41.363917 (+ 527us) server_negotiation.cc:299] Negotiation successful
0924 19:05:41.364286 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":534,"thread_start_us":370,"threads_started":1}
I20260924 19:05:41.369674 23209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36443 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:05:41.370707 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:41.370323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36443 (local address 127.0.0.1:35870)
0924 19:05:41.370488 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:41.370600 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260924 19:05:41.370982 23160 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36443: connect: Connection refused (error 111)
I20260924 19:05:41.372519 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:41.372193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36443 (local address 127.0.0.1:35878)
0924 19:05:41.372330 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:41.372397 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 19:05:41.372732 23209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36443: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36443: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260924 19:05:41.392225 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36443
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38773,127.0.97.61:36443,127.0.97.60:45101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:05:41.422034 23209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36443 } attempt: 1
I20260924 19:05:41.423199 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:41.422685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36443 (local address 127.0.0.1:35880)
0924 19:05:41.422911 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:41.423039 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 19:05:41.424242 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:41.423845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36443 (local address 127.0.0.1:35894)
0924 19:05:41.424008 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:41.424092 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 19:05:41.424458 23209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36443: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36443: connect: Connection refused (error 111)
I20260924 19:05:41.492703 23209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36443 } attempt: 2
I20260924 19:05:41.493988 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:41.493381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36443 (local address 127.0.0.1:35896)
0924 19:05:41.493684 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:41.493795 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 19:05:41.495173 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:41.494710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36443 (local address 127.0.0.1:35902)
0924 19:05:41.494914 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:41.495012 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 19:05:41.495384 23209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36443: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36443: connect: Connection refused (error 111)
I20260924 19:05:41.586656 23209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36443 } attempt: 3
I20260924 19:05:41.587886 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:41.587358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36443 (local address 127.0.0.1:35908)
0924 19:05:41.587598 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:41.587709 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 19:05:41.589062 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:41.588658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36443 (local address 127.0.0.1:35918)
0924 19:05:41.588819 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:41.588901 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 19:05:41.589280 23209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36443: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36443: connect: Connection refused (error 111)
W20260924 19:05:41.714893 23216 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:05:41.715242 23216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:41.715368 23216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:41.715519 23216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:41.722616 23216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:05:41.722699 23216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:41.722827 23216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:41.722910 23216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:05:41.722993 23216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:05:41.732529 23209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36443 } attempt: 4
I20260924 19:05:41.732558 23216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38773,127.0.97.61:36443,127.0.97.60:45101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:41.733904 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:41.733132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36443 (local address 127.0.0.1:35922)
0924 19:05:41.733577 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:41.733689 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233}
I20260924 19:05:41.734328 23216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:41.735020 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:41.734622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36443 (local address 127.0.0.1:35936)
0924 19:05:41.734775 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:41.734908 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 19:05:41.735275 23209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36443: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36443: connect: Connection refused (error 111)
I20260924 19:05:41.735925 23216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:41.748224 23221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:41.748157 23224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:41.748262 23222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:41.749200 23216 server_base.cc:1034] running on GCE node
I20260924 19:05:41.749994 23216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:41.751394 23216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:41.752635 23216 hybrid_clock.cc:648] HybridClock initialized: now 1790276741752577 us; error 42 us; skew 500 ppm
I20260924 19:05:41.752952 23216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:41 dist-test-slave-9fn1 krb5kdc[23129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:41.756012 23216 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260924 19:05:41.757403 23216 webserver.cc:460] Webserver started at http://127.0.97.61:34465/ using document root <none> and password file <none>
I20260924 19:05:41.758098 23216 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:41.758183 23216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:41.758432 23216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:41.761145 23216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-1/data/instance:
uuid: "1f6f82558166458083b8913ec15e6133"
format_stamp: "Formatted at 2026-09-24 19:05:41 on dist-test-slave-9fn1"
server_key: "bce23e7d47cb056d7675085fd2fc0f24"
server_key_iv: "70dbfd9469c7f4df4fee878dbd6e5e97"
server_key_version: "encryptionkey@0"
I20260924 19:05:41.761909 23216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-1/wal/instance:
uuid: "1f6f82558166458083b8913ec15e6133"
format_stamp: "Formatted at 2026-09-24 19:05:41 on dist-test-slave-9fn1"
server_key: "bce23e7d47cb056d7675085fd2fc0f24"
server_key_iv: "70dbfd9469c7f4df4fee878dbd6e5e97"
server_key_version: "encryptionkey@0"
I20260924 19:05:41.766631 23216 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 19:05:41.769784 23231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:41.771031 23216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:05:41.771219 23216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-1/wal
uuid: "1f6f82558166458083b8913ec15e6133"
format_stamp: "Formatted at 2026-09-24 19:05:41 on dist-test-slave-9fn1"
server_key: "bce23e7d47cb056d7675085fd2fc0f24"
server_key_iv: "70dbfd9469c7f4df4fee878dbd6e5e97"
server_key_version: "encryptionkey@0"
I20260924 19:05:41.771412 23216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:41.805233 23216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:41.809592 23216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:41.809860 23216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:41.828570 23216 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:36443
I20260924 19:05:41.828595 23282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:36443 every 8 connection(s)
I20260924 19:05:41.830008 23216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-1/data/info.pb
I20260924 19:05:41.833838 23283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:05:41.836403 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 23216
I20260924 19:05:41.836577 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-1/wal/instance
I20260924 19:05:41.837036 388 external_mini_cluster.cc:1428] Setting key 96c814576de12f475c5f2275f8d6250e
I20260924 19:05:41.840489 23283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:05:41 dist-test-slave-9fn1 krb5kdc[23129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260924 19:05:41.861290 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:41.843234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48750 (local address 127.0.97.62:38773)
0924 19:05:41.843425 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:41.843429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:41.844667 (+ 1238us) server_negotiation.cc:407] Connection header received
0924 19:05:41.846987 (+ 2320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:41.846995 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:41.847089 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:41.847199 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:41.848644 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:41.849349 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:41.850854 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:41.851255 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:41.851397 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:41.855332 (+ 3935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:41.855337 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:41.855341 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:41.855372 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:41.858191 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:41.859153 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:41.859156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:41.859159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:41.859270 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:41.859746 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:41.859749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:41.859752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:41.860027 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:41.860137 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:41.860886 (+ 749us) server_negotiation.cc:299] Negotiation successful
0924 19:05:41.861056 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260924 19:05:41.862596 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:41.843733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38773 (local address 127.0.0.1:48750)
0924 19:05:41.844498 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:41.844541 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:41.846729 (+ 2188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:41.847434 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:41.847443 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:41.847931 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:05:41.848459 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:41.848475 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:41.849529 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:41.849536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:41.850694 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:41.850705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:41.851480 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:41.851487 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:41.851711 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:41.852393 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:05:41.852415 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:05:41.855115 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:05:41.858333 (+ 3218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:41.858343 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:41.858359 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:41.859025 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:41.859378 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:41.859383 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:41.859387 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:41.859628 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:41.860146 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:05:41.860151 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:05:41.860485 (+ 334us) client_negotiation.cc:769] Sending connection context
0924 19:05:41.860759 (+ 274us) client_negotiation.cc:240] Negotiation successful
0924 19:05:41.861370 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":340,"threads_started":1}
I20260924 19:05:41.864464 23283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36443 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:05:41 dist-test-slave-9fn1 krb5kdc[23129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 19:05:41.879587 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:41.865338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36443 (local address 127.0.0.1:35944)
0924 19:05:41.865490 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:41.865570 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:41.865665 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:41.866652 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:41.866656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:41.867082 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:05:41.867203 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:41.867211 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:41.868507 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:41.868513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:41.869359 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:41.869368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:41.870171 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:41.870176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:41.870347 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:41.870930 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:05:41.870946 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:05:41.873039 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:05:41.877254 (+ 4215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:41.877258 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:41.877261 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:41.877676 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:41.878073 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:41.878076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:41.878078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:41.878210 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:41.878946 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:05:41.878949 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:05:41.879056 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 19:05:41.879160 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 19:05:41.879346 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 19:05:41.880095 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:41.865660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35944 (local address 127.0.97.61:36443)
0924 19:05:41.866271 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:41.866277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:41.866300 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:05:41.866349 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:41.866354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:41.866514 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:41.866667 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:41.867372 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:41.868363 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:41.869491 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:41.870038 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:41.870168 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:41.873236 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:41.873247 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:41.873259 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:41.873285 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:41.877121 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:41.877807 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:41.877818 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:41.877824 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:41.877966 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:41.878312 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:41.878319 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:41.878325 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:41.878810 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:41.879033 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:41.879337 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 19:05:41.879608 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":355,"threads_started":1}
I20260924 19:05:41.882550 23283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:05:41.883667 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:41.883232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45101 (local address 127.0.0.1:35680)
0924 19:05:41.883421 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:41.883551 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 19:05:41.883912 23235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45101: connect: Connection refused (error 111)
I20260924 19:05:41.885330 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:41.885036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45101 (local address 127.0.0.1:35686)
0924 19:05:41.885175 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:41.885242 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 19:05:41.885543 23283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45101: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45101: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260924 19:05:41.895006 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLphABj/build/asan/bin/kudu
/tmp/dist-test-taskLphABj/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45101
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38773,127.0.97.61:36443,127.0.97.60:45101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:05:41.934844 23283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } attempt: 1
I20260924 19:05:41.935943 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:41.935492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45101 (local address 127.0.0.1:35692)
0924 19:05:41.935679 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:41.935772 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 19:05:41.936872 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:41.936544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45101 (local address 127.0.0.1:35698)
0924 19:05:41.936699 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:41.936765 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 19:05:41.937060 23283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45101: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45101: connect: Connection refused (error 111)
I20260924 19:05:42.005244 23283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } attempt: 2
I20260924 19:05:42.006284 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.005863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45101 (local address 127.0.0.1:35704)
0924 19:05:42.006050 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:42.006139 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 19:05:42.007162 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.006860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45101 (local address 127.0.0.1:35716)
0924 19:05:42.006992 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:42.007058 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 19:05:42.007359 23283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45101: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45101: connect: Connection refused (error 111)
I20260924 19:05:42.034535 23209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36443 } attempt: 5
Sep 24 19:05:42 dist-test-slave-9fn1 krb5kdc[23129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 19:05:42.049149 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.035323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36443 (local address 127.0.0.1:35948)
0924 19:05:42.035540 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:42.035563 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:42.035689 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:42.036057 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:42.036060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:42.036452 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:05:42.036617 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:42.036625 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.038174 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.038178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:42.038971 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:42.038988 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.039886 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.039890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:42.040037 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:42.040637 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:05:42.040653 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:05:42.042949 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:05:42.046902 (+ 3953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:42.046906 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:42.046909 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:42.047260 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:42.047632 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:42.047635 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:42.047637 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:42.047768 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:42.048480 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:05:42.048483 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:05:42.048602 (+ 119us) client_negotiation.cc:769] Sending connection context
0924 19:05:42.048708 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 19:05:42.048911 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 19:05:42.049606 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.035514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35948 (local address 127.0.97.61:36443)
0924 19:05:42.035715 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:42.035721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:42.035743 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:05:42.035807 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:42.035812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:42.035907 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:42.036042 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:42.036772 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.038036 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.039125 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.039749 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.039866 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:42.043140 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:42.043147 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:42.043151 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:42.043188 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:42.046776 (+ 3588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:42.047373 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:42.047378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:42.047382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:42.047527 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:42.047898 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:42.047905 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:42.047911 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:42.048355 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:42.048526 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:42.048877 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 19:05:42.049156 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:05:42.050632 23209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:05:42.051828 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.051320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45101 (local address 127.0.0.1:35732)
0924 19:05:42.051522 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:42.051630 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 19:05:42.052981 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.052533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45101 (local address 127.0.0.1:35740)
0924 19:05:42.052709 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:42.052825 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 19:05:42.053251 23209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45101: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45101: connect: Connection refused (error 111)
I20260924 19:05:42.098553 23283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } attempt: 3
I20260924 19:05:42.099668 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.099195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45101 (local address 127.0.0.1:35756)
0924 19:05:42.099428 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:42.099512 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 19:05:42.100786 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.100399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45101 (local address 127.0.0.1:35758)
0924 19:05:42.100616 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:42.100682 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260924 19:05:42.101079 23283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45101: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45101: connect: Connection refused (error 111)
I20260924 19:05:42.104501 23209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } attempt: 1
I20260924 19:05:42.105561 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.105067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45101 (local address 127.0.0.1:35768)
0924 19:05:42.105251 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:42.105345 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 19:05:42.106554 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.106182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45101 (local address 127.0.0.1:35774)
0924 19:05:42.106328 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:42.106431 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 19:05:42.106805 23209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45101: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45101: connect: Connection refused (error 111)
I20260924 19:05:42.175025 23209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } attempt: 2
I20260924 19:05:42.176303 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.175729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45101 (local address 127.0.0.1:35788)
0924 19:05:42.175969 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:42.176090 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 19:05:42.177335 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.176943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45101 (local address 127.0.0.1:35792)
0924 19:05:42.177119 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:42.177206 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 19:05:42.177583 23209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45101: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45101: connect: Connection refused (error 111)
W20260924 19:05:42.223881 23290 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:05:42.224176 23290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:42.224247 23290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:42.224325 23290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:42.231359 23290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:05:42.231436 23290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:42.231492 23290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:42.231547 23290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:05:42.231608 23290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:05:42.240662 23290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38773,127.0.97.61:36443,127.0.97.60:45101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:42.241804 23290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:42.243010 23290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:05:42.244290 23283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } attempt: 4
I20260924 19:05:42.245630 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.244970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45101 (local address 127.0.0.1:35796)
0924 19:05:42.245251 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:42.245378 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 19:05:42.246629 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.246311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45101 (local address 127.0.0.1:35808)
0924 19:05:42.246468 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:42.246530 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 19:05:42.246874 23283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45101: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45101: connect: Connection refused (error 111)
W20260924 19:05:42.254823 23295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:42.254916 23298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:42.254879 23296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:42.255793 23290 server_base.cc:1034] running on GCE node
I20260924 19:05:42.256623 23290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:42.258056 23290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:42.259267 23290 hybrid_clock.cc:648] HybridClock initialized: now 1790276742259193 us; error 50 us; skew 500 ppm
I20260924 19:05:42.259588 23290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:42 dist-test-slave-9fn1 krb5kdc[23129](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:42.262660 23290 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260924 19:05:42.264030 23290 webserver.cc:460] Webserver started at http://127.0.97.60:36071/ using document root <none> and password file <none>
I20260924 19:05:42.264648 23290 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:42.264730 23290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:42.264988 23290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:42.267868 23290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-2/data/instance:
uuid: "4a3bec28013f43878f2e7736da916870"
format_stamp: "Formatted at 2026-09-24 19:05:42 on dist-test-slave-9fn1"
server_key: "1572d1411bfb63da4b7c728343d45106"
server_key_iv: "34adc7ceb8caa20aef7549619ddde4a9"
server_key_version: "encryptionkey@0"
I20260924 19:05:42.268683 23290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-2/wal/instance:
uuid: "4a3bec28013f43878f2e7736da916870"
format_stamp: "Formatted at 2026-09-24 19:05:42 on dist-test-slave-9fn1"
server_key: "1572d1411bfb63da4b7c728343d45106"
server_key_iv: "34adc7ceb8caa20aef7549619ddde4a9"
server_key_version: "encryptionkey@0"
I20260924 19:05:42.273572 23290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260924 19:05:42.276958 23305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:42.278395 23290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 19:05:42.278565 23290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-2/wal
uuid: "4a3bec28013f43878f2e7736da916870"
format_stamp: "Formatted at 2026-09-24 19:05:42 on dist-test-slave-9fn1"
server_key: "1572d1411bfb63da4b7c728343d45106"
server_key_iv: "34adc7ceb8caa20aef7549619ddde4a9"
server_key_version: "encryptionkey@0"
I20260924 19:05:42.278771 23290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:42.283821 23209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } attempt: 3
I20260924 19:05:42.284929 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.284440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45101 (local address 127.0.0.1:35824)
0924 19:05:42.284653 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:42.284753 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 19:05:42.285969 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.285617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45101 (local address 127.0.0.1:35826)
0924 19:05:42.285783 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:42.285864 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260924 19:05:42.286180 23209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45101: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45101: connect: Connection refused (error 111)
I20260924 19:05:42.310590 23290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:42.314363 23290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:42.314644 23290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:42.332864 23290 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:45101
I20260924 19:05:42.332887 23356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:45101 every 8 connection(s)
I20260924 19:05:42.334265 23290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-2/data/info.pb
I20260924 19:05:42.338028 23357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:05:42.339483 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLphABj/build/asan/bin/kudu as pid 23290
I20260924 19:05:42.339664 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLphABj/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276386554690-388-0/minicluster-data/master-2/wal/instance
I20260924 19:05:42.340101 388 external_mini_cluster.cc:1428] Setting key 3f58fb6b31d149f0615658a969fe7b2c
I20260924 19:05:42.347285 23357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:05:42 dist-test-slave-9fn1 krb5kdc[23129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 24 19:05:42 dist-test-slave-9fn1 krb5kdc[23129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260924 19:05:42.361725 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.341840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35832 (local address 127.0.97.60:45101)
0924 19:05:42.342651 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:42.342662 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:42.342705 (+ 43us) server_negotiation.cc:407] Connection header received
0924 19:05:42.344212 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:42.344233 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:42.344628 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:42.344946 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:42.346677 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.347934 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.349218 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.350114 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.350306 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:42.354382 (+ 4076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:42.354390 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:42.354400 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:42.354462 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:42.357761 (+ 3299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:42.358533 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:42.358541 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:42.358551 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:42.358729 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:42.359142 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:42.359145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:42.359147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:42.359611 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:42.359830 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:42.360293 (+ 463us) server_negotiation.cc:299] Negotiation successful
0924 19:05:42.360692 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":405,"threads_started":1}
I20260924 19:05:42.365065 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.349827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38773 (local address 127.0.0.1:48764)
0924 19:05:42.350632 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:42.350678 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:42.350871 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:42.351263 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:42.351269 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:42.351842 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:05:42.352074 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:42.352087 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.353206 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.353209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:42.354344 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:42.354356 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.355384 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.355391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:42.355538 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:42.356158 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:05:42.356185 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:05:42.358495 (+ 2310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:05:42.362026 (+ 3531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:42.362037 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:42.362042 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:42.362635 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:42.363283 (+ 648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:42.363288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:42.363291 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:42.363477 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:42.363968 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:05:42.363981 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:05:42.364164 (+ 183us) client_negotiation.cc:769] Sending connection context
0924 19:05:42.364365 (+ 201us) client_negotiation.cc:240] Negotiation successful
0924 19:05:42.364659 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":430,"threads_started":1}
I20260924 19:05:42.365604 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.349816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48764 (local address 127.0.97.62:38773)
0924 19:05:42.350046 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:42.350054 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:42.350761 (+ 707us) server_negotiation.cc:407] Connection header received
0924 19:05:42.351022 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:42.351028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:42.351101 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:42.351205 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:42.352223 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.353046 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.354484 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.355173 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.355318 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:42.358679 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:42.358684 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:42.358687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:42.358718 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:42.361861 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:42.362797 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:42.362805 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:42.362811 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:42.363118 (+ 307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:42.363584 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:42.363587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:42.363589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:42.363846 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:42.363968 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:42.364499 (+ 531us) server_negotiation.cc:299] Negotiation successful
0924 19:05:42.365172 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 19:05:42.366961 23357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36443 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:05:42 dist-test-slave-9fn1 krb5kdc[23129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 24 19:05:42 dist-test-slave-9fn1 krb5kdc[23129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 19:05:42.380867 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.364951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48776 (local address 127.0.97.62:38773)
0924 19:05:42.366147 (+ 1196us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:42.366152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:42.366169 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:05:42.366212 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:42.366217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:42.366305 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:42.366407 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:42.367870 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.369308 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.370938 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.371567 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.371693 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:42.375919 (+ 4226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:42.375927 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:42.375935 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:42.375976 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:42.378494 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:42.379177 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:42.379182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:42.379185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:42.379332 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:42.379655 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:42.379662 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:42.379668 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:42.379960 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:42.380245 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:42.380433 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 19:05:42.380580 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1112,"thread_start_us":3012,"threads_started":1}
I20260924 19:05:42.382025 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.367836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36443 (local address 127.0.0.1:35960)
0924 19:05:42.368191 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:42.368210 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:42.368300 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:42.368656 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:42.368661 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:42.369112 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:05:42.369268 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:42.369278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.370624 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.370627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:42.371876 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:42.371886 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.372783 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.372788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:42.372921 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:42.373426 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:05:42.373441 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:05:42.375863 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:05:42.379178 (+ 3315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:42.379182 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:42.379186 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:42.379634 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:42.379997 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:42.380004 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:42.380009 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:42.380198 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:42.380795 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:05:42.380799 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:05:42.380949 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 19:05:42.381398 (+ 449us) client_negotiation.cc:240] Negotiation successful
0924 19:05:42.381689 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249}
I20260924 19:05:42.382382 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.368234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35960 (local address 127.0.97.61:36443)
0924 19:05:42.368398 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:42.368403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:42.368417 (+ 14us) server_negotiation.cc:407] Connection header received
0924 19:05:42.368464 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:42.368467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:42.368520 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:42.368597 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:42.369436 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.370489 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.372013 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.372631 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.372758 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:42.376023 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:42.376028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:42.376031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:42.376061 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:42.379032 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:42.379738 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:42.379742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:42.379744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:42.379867 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:42.380316 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:42.380324 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:42.380329 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:42.380656 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:42.380803 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:42.381548 (+ 745us) server_negotiation.cc:299] Negotiation successful
0924 19:05:42.381803 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 19:05:42.383512 23357 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:05:42 dist-test-slave-9fn1 krb5kdc[23129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 24 19:05:42 dist-test-slave-9fn1 krb5kdc[23129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260924 19:05:42.396590 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.384212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45101 (local address 127.0.0.1:35846)
0924 19:05:42.384370 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:42.384397 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:42.384501 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:42.384795 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:42.384798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:42.385102 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:05:42.385225 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:42.385232 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.386241 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.386246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:42.387320 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:42.387331 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.388057 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.388064 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:42.388257 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:42.388755 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:05:42.388770 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:05:42.390926 (+ 2156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:05:42.394279 (+ 3353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:42.394287 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:42.394291 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:42.394632 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:42.394973 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:42.394980 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:42.394983 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:42.395118 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:42.395594 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:05:42.395599 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:05:42.395728 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 19:05:42.395859 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 19:05:42.396148 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 19:05:42.396631 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.384244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35846 (local address 127.0.97.60:45101)
0924 19:05:42.384394 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:42.384398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:42.384447 (+ 49us) server_negotiation.cc:407] Connection header received
0924 19:05:42.384599 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:42.384604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:42.384674 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:42.384771 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:42.385355 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.386120 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.387499 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.387893 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.387986 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:42.391091 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:42.391100 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:42.391105 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:42.391141 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:42.394156 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:42.394752 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:42.394759 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:42.394762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:42.394870 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:42.395214 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:42.395222 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:42.395225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:42.395478 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:42.395618 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:42.395947 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 19:05:42.396288 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":54}
I20260924 19:05:42.397420 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.382606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35970 (local address 127.0.97.61:36443)
0924 19:05:42.382990 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:42.382995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:42.383019 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:05:42.383056 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:42.383059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:42.383134 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:42.383232 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:42.384613 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.385475 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.386893 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.387300 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.387392 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:42.391977 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:42.391983 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:42.391988 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:42.392026 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:42.394452 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:42.395125 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:42.395132 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:42.395135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:42.395270 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:42.395651 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:42.395656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:42.395659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:42.395909 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:42.396175 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:42.396467 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 19:05:42.397156 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279}
I20260924 19:05:42.401851 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:05:42.401851 23357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a3bec28013f43878f2e7736da916870: Bootstrap starting.
I20260924 19:05:42.404994 23357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a3bec28013f43878f2e7736da916870: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:42.406091 23357 log.cc:824] T 00000000000000000000000000000000 P 4a3bec28013f43878f2e7736da916870: Log is configured to *not* fsync() on all Append() calls
I20260924 19:05:42.404132 23184 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c8493804eaad4c1fba9431b9e2c317d0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48776
I20260924 19:05:42.408725 23357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a3bec28013f43878f2e7736da916870: No bootstrap required, opened a new log
I20260924 19:05:42.415639 23357 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a3bec28013f43878f2e7736da916870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8493804eaad4c1fba9431b9e2c317d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "1f6f82558166458083b8913ec15e6133" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36443 } } peers { permanent_uuid: "4a3bec28013f43878f2e7736da916870" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } }
I20260924 19:05:42.415962 23357 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a3bec28013f43878f2e7736da916870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a3bec28013f43878f2e7736da916870, State: Initialized, Role: FOLLOWER
I20260924 19:05:42.416519 23357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a3bec28013f43878f2e7736da916870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8493804eaad4c1fba9431b9e2c317d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "1f6f82558166458083b8913ec15e6133" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36443 } } peers { permanent_uuid: "4a3bec28013f43878f2e7736da916870" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } }
I20260924 19:05:42.417404 23364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a3bec28013f43878f2e7736da916870 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8493804eaad4c1fba9431b9e2c317d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "1f6f82558166458083b8913ec15e6133" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36443 } } peers { permanent_uuid: "4a3bec28013f43878f2e7736da916870" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } } }
I20260924 19:05:42.417771 23364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a3bec28013f43878f2e7736da916870 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:05:42.418628 23357 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a3bec28013f43878f2e7736da916870 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 19:05:42.427699 23375 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4a3bec28013f43878f2e7736da916870: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:05:42.427827 23375 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:05:42.425567 23290 master_runner.cc:186] Error getting master registration for 127.0.97.62:38773: OK, instance_id { permanent_uuid: "c8493804eaad4c1fba9431b9e2c317d0" instance_seqno: 1790276741327152 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:05:42.436661 23290 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:05:42.463366 23209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } attempt: 4
Sep 24 19:05:42 dist-test-slave-9fn1 krb5kdc[23129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260924 19:05:42.479616 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.464107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45101 (local address 127.0.0.1:35858)
0924 19:05:42.464324 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:42.464347 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:42.464480 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:42.464796 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:42.464801 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:42.465363 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:05:42.465626 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:42.465640 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.467110 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.467116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:42.468567 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:42.468583 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.469606 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.469613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:42.469827 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:42.470534 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:05:42.470558 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:05:42.473311 (+ 2753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:05:42.477117 (+ 3806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:42.477122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:42.477126 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:42.477474 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:42.477859 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:42.477865 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:42.477869 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:42.478011 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:42.478641 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:05:42.478647 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:05:42.478798 (+ 151us) client_negotiation.cc:769] Sending connection context
0924 19:05:42.478940 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 19:05:42.479237 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 19:05:42.479622 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.464216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35858 (local address 127.0.97.60:45101)
0924 19:05:42.464425 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:42.464430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:42.464449 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:05:42.464603 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:42.464607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:42.464665 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:42.464770 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:42.465778 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.466935 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.468709 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.469411 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.469555 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:42.473529 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:42.473535 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:42.473540 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:42.473574 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:42.476998 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:42.477613 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:42.477621 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:42.477626 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:42.477764 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:42.478132 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:42.478140 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:42.478144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:42.478529 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:42.478678 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:42.479048 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 19:05:42.479259 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 19:05:42.485461 23209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8493804eaad4c1fba9431b9e2c317d0: Bootstrap starting.
I20260924 19:05:42.489459 23209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8493804eaad4c1fba9431b9e2c317d0: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:42.490749 23209 log.cc:824] T 00000000000000000000000000000000 P c8493804eaad4c1fba9431b9e2c317d0: Log is configured to *not* fsync() on all Append() calls
I20260924 19:05:42.494267 23209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8493804eaad4c1fba9431b9e2c317d0: No bootstrap required, opened a new log
I20260924 19:05:42.501719 23209 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8493804eaad4c1fba9431b9e2c317d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8493804eaad4c1fba9431b9e2c317d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "1f6f82558166458083b8913ec15e6133" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36443 } } peers { permanent_uuid: "4a3bec28013f43878f2e7736da916870" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } }
I20260924 19:05:42.502120 23209 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8493804eaad4c1fba9431b9e2c317d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8493804eaad4c1fba9431b9e2c317d0, State: Initialized, Role: FOLLOWER
I20260924 19:05:42.502622 23209 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8493804eaad4c1fba9431b9e2c317d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8493804eaad4c1fba9431b9e2c317d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "1f6f82558166458083b8913ec15e6133" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36443 } } peers { permanent_uuid: "4a3bec28013f43878f2e7736da916870" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } }
I20260924 19:05:42.503612 23378 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8493804eaad4c1fba9431b9e2c317d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8493804eaad4c1fba9431b9e2c317d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "1f6f82558166458083b8913ec15e6133" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36443 } } peers { permanent_uuid: "4a3bec28013f43878f2e7736da916870" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } } }
I20260924 19:05:42.503989 23378 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8493804eaad4c1fba9431b9e2c317d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:05:42.504882 23209 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8493804eaad4c1fba9431b9e2c317d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:05:42.504937 23184 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8493804eaad4c1fba9431b9e2c317d0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 19:05:42.505160 23184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8493804eaad4c1fba9431b9e2c317d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:42.507699 23184 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8493804eaad4c1fba9431b9e2c317d0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8493804eaad4c1fba9431b9e2c317d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "1f6f82558166458083b8913ec15e6133" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36443 } } peers { permanent_uuid: "4a3bec28013f43878f2e7736da916870" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } }
I20260924 19:05:42.509840 23332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8493804eaad4c1fba9431b9e2c317d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4a3bec28013f43878f2e7736da916870"
I20260924 19:05:42.510360 23332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a3bec28013f43878f2e7736da916870 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:42.513700 23332 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4a3bec28013f43878f2e7736da916870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8493804eaad4c1fba9431b9e2c317d0 in term 1.
I20260924 19:05:42.514068 23184 leader_election.cc:288] T 00000000000000000000000000000000 P c8493804eaad4c1fba9431b9e2c317d0 [CANDIDATE]: Term 1 election: Requested vote from peers 1f6f82558166458083b8913ec15e6133 (127.0.97.61:36443), 4a3bec28013f43878f2e7736da916870 (127.0.97.60:45101)
I20260924 19:05:42.514361 23161 leader_election.cc:302] T 00000000000000000000000000000000 P c8493804eaad4c1fba9431b9e2c317d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a3bec28013f43878f2e7736da916870, c8493804eaad4c1fba9431b9e2c317d0; no voters:
I20260924 19:05:42.514834 23378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8493804eaad4c1fba9431b9e2c317d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:05:42.546073 23283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } attempt: 5
Sep 24 19:05:42 dist-test-slave-9fn1 krb5kdc[23129](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
W20260924 19:05:42.566697 23311 tablet.cc:2369] T 00000000000000000000000000000000 P 4a3bec28013f43878f2e7736da916870: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:05:42.588146 23163 tablet.cc:2369] T 00000000000000000000000000000000 P c8493804eaad4c1fba9431b9e2c317d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:05:42.589896 23378 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8493804eaad4c1fba9431b9e2c317d0 [term 1 LEADER]: Becoming Leader. State: Replica: c8493804eaad4c1fba9431b9e2c317d0, State: Running, Role: LEADER
I20260924 19:05:42.594197 23378 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8493804eaad4c1fba9431b9e2c317d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8493804eaad4c1fba9431b9e2c317d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "1f6f82558166458083b8913ec15e6133" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36443 } } peers { permanent_uuid: "4a3bec28013f43878f2e7736da916870" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } }
I20260924 19:05:42.595566 23285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.546814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45101 (local address 127.0.0.1:35870)
0924 19:05:42.547312 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:42.547337 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:42.547494 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:42.548588 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:42.548596 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:42.549297 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:05:42.549599 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:42.549614 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.551599 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.551607 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:42.553553 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:42.553574 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.555223 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.555232 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:42.555567 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:42.556445 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:05:42.556477 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:05:42.562326 (+ 5849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:05:42.570733 (+ 8407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:42.570742 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:42.570772 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:42.571982 (+ 1210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:42.574882 (+ 2900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:42.574891 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:42.574898 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:42.575368 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:42.590375 (+ 15007us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:05:42.590385 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:05:42.593589 (+ 3204us) client_negotiation.cc:769] Sending connection context
0924 19:05:42.593800 (+ 211us) client_negotiation.cc:240] Negotiation successful
0924 19:05:42.595171 (+ 1371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345}
I20260924 19:05:42.596696 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.547776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35870 (local address 127.0.97.60:45101)
0924 19:05:42.547994 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:42.548002 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:42.548024 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:05:42.548104 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:42.548111 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:42.548216 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:42.548336 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:42.549843 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.551353 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.553802 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.554752 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.554905 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:42.562757 (+ 7852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:42.562768 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:42.562776 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:42.562836 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:42.570290 (+ 7454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:42.572995 (+ 2705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:42.573033 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:42.573041 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:42.573456 (+ 415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:42.576185 (+ 2729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:42.576222 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:42.576229 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:42.577173 (+ 944us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:42.577550 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:42.594531 (+ 16981us) server_negotiation.cc:299] Negotiation successful
0924 19:05:42.596315 (+ 1784us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 19:05:42.603928 23283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f6f82558166458083b8913ec15e6133: Bootstrap starting.
I20260924 19:05:42.609182 23283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f6f82558166458083b8913ec15e6133: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:42.610965 23283 log.cc:824] T 00000000000000000000000000000000 P 1f6f82558166458083b8913ec15e6133: Log is configured to *not* fsync() on all Append() calls
I20260924 19:05:42.625762 23390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.579218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35872 (local address 127.0.97.60:45101)
0924 19:05:42.592737 (+ 13519us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:42.592744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:42.598167 (+ 5423us) server_negotiation.cc:407] Connection header received
0924 19:05:42.598232 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:42.598238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:42.598314 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:42.598411 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:42.600708 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.601894 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.607343 (+ 5449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.608200 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.608352 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:42.617733 (+ 9381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:42.617744 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:42.617752 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:42.617808 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:42.621811 (+ 4003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:42.622703 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:42.622712 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:42.622717 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:42.622874 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:42.623313 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:42.623318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:42.623321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:42.623707 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:42.623956 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:42.624752 (+ 796us) server_negotiation.cc:299] Negotiation successful
0924 19:05:42.624977 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13344,"thread_start_us":13113,"threads_started":1}
I20260924 19:05:42.627753 23383 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8493804eaad4c1fba9431b9e2c317d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8493804eaad4c1fba9431b9e2c317d0. Latest consensus state: current_term: 1 leader_uuid: "c8493804eaad4c1fba9431b9e2c317d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8493804eaad4c1fba9431b9e2c317d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "1f6f82558166458083b8913ec15e6133" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36443 } } peers { permanent_uuid: "4a3bec28013f43878f2e7736da916870" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } } }
I20260924 19:05:42.628322 23383 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8493804eaad4c1fba9431b9e2c317d0 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:42.635136 23401 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 19:05:42.636514 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.538713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48786 (local address 127.0.97.62:38773)
0924 19:05:42.538966 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:42.538975 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:42.593010 (+ 54035us) server_negotiation.cc:407] Connection header received
0924 19:05:42.595520 (+ 2510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:42.595527 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:42.595614 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:42.595686 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:42.600656 (+ 4970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.601841 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.608805 (+ 6964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.609673 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.609793 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:42.620569 (+ 10776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:42.620577 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:42.620582 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:42.620632 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:42.624308 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:42.630925 (+ 6617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:42.630931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:42.630935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:42.631138 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:42.633035 (+ 1897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:42.633041 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:42.633044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:42.633395 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:42.633679 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:42.635557 (+ 1878us) server_negotiation.cc:299] Negotiation successful
0924 19:05:42.635823 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260924 19:05:42.652402 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:42.540895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35986 (local address 127.0.97.61:36443)
0924 19:05:42.541146 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:42.541155 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:42.593490 (+ 52335us) server_negotiation.cc:407] Connection header received
0924 19:05:42.593740 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:42.593746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:42.593833 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:42.593932 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:42.604596 (+ 10664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.605749 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.612367 (+ 6618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:42.613024 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:42.613135 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:42.625647 (+ 12512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:42.625654 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:42.625659 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:42.625711 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:42.629337 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:42.634198 (+ 4861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:42.634204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:42.634208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:42.634377 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:42.650262 (+ 15885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:42.650272 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:42.650279 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:42.650746 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:42.651024 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:42.651703 (+ 679us) server_negotiation.cc:299] Negotiation successful
0924 19:05:42.651984 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":817,"server-negotiator.queue_time_us":64}
I20260924 19:05:42.653798 23398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:05:42.639448 23142 master_runner.cc:186] Error getting master registration for 127.0.97.61:36443: OK, instance_id { permanent_uuid: "1f6f82558166458083b8913ec15e6133" instance_seqno: 1790276741823251 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:05:42.658263 23142 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:05:42.663632 23283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f6f82558166458083b8913ec15e6133: No bootstrap required, opened a new log
I20260924 19:05:42.669991 23398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:05:42.674932 23283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f6f82558166458083b8913ec15e6133 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8493804eaad4c1fba9431b9e2c317d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "1f6f82558166458083b8913ec15e6133" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36443 } } peers { permanent_uuid: "4a3bec28013f43878f2e7736da916870" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } }
I20260924 19:05:42.675326 23283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f6f82558166458083b8913ec15e6133 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6f82558166458083b8913ec15e6133, State: Initialized, Role: FOLLOWER
I20260924 19:05:42.675875 23283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f6f82558166458083b8913ec15e6133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8493804eaad4c1fba9431b9e2c317d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "1f6f82558166458083b8913ec15e6133" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36443 } } peers { permanent_uuid: "4a3bec28013f43878f2e7736da916870" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } }
I20260924 19:05:42.691591 23283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f6f82558166458083b8913ec15e6133 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:05:42.691660 23258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c8493804eaad4c1fba9431b9e2c317d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1f6f82558166458083b8913ec15e6133"
I20260924 19:05:42.692162 23258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f6f82558166458083b8913ec15e6133 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:42.696585 23258 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1f6f82558166458083b8913ec15e6133 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8493804eaad4c1fba9431b9e2c317d0 in term 1.
I20260924 19:05:42.691282 23403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f6f82558166458083b8913ec15e6133 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8493804eaad4c1fba9431b9e2c317d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "1f6f82558166458083b8913ec15e6133" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36443 } } peers { permanent_uuid: "4a3bec28013f43878f2e7736da916870" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } } }
I20260924 19:05:42.701669 23403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f6f82558166458083b8913ec15e6133 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:05:42.705260 23332 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4a3bec28013f43878f2e7736da916870 [term 1 FOLLOWER]: Refusing update from remote peer c8493804eaad4c1fba9431b9e2c317d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:05:42.709323 23378 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8493804eaad4c1fba9431b9e2c317d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a3bec28013f43878f2e7736da916870" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:05:42.709986 23258 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1f6f82558166458083b8913ec15e6133 [term 1 FOLLOWER]: Refusing update from remote peer c8493804eaad4c1fba9431b9e2c317d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:05:42.713800 23378 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c8493804eaad4c1fba9431b9e2c317d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f6f82558166458083b8913ec15e6133" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:05:42.723272 23364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a3bec28013f43878f2e7736da916870 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8493804eaad4c1fba9431b9e2c317d0. Latest consensus state: current_term: 1 leader_uuid: "c8493804eaad4c1fba9431b9e2c317d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8493804eaad4c1fba9431b9e2c317d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "1f6f82558166458083b8913ec15e6133" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36443 } } peers { permanent_uuid: "4a3bec28013f43878f2e7736da916870" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } } }
I20260924 19:05:42.723595 23364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a3bec28013f43878f2e7736da916870 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:05:42.728677 23404 mvcc.cc:205] Tried to move back new op lower bound from 7332973538110406656 to 7332973537676238848. Current Snapshot: MvccSnapshot[applied={T|T < 7332973538110406656}]
I20260924 19:05:42.730037 23378 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8493804eaad4c1fba9431b9e2c317d0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8493804eaad4c1fba9431b9e2c317d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8493804eaad4c1fba9431b9e2c317d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "1f6f82558166458083b8913ec15e6133" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36443 } } peers { permanent_uuid: "4a3bec28013f43878f2e7736da916870" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } } }
I20260924 19:05:42.730360 23378 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8493804eaad4c1fba9431b9e2c317d0 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:42.731675 23398 catalog_manager.cc:1347] Generated new cluster ID: 70cedae9573943ac8b0592112f9f6cd7
I20260924 19:05:42.731786 23398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:05:42.734884 23403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f6f82558166458083b8913ec15e6133 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8493804eaad4c1fba9431b9e2c317d0. Latest consensus state: current_term: 1 leader_uuid: "c8493804eaad4c1fba9431b9e2c317d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8493804eaad4c1fba9431b9e2c317d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "1f6f82558166458083b8913ec15e6133" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36443 } } peers { permanent_uuid: "4a3bec28013f43878f2e7736da916870" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } } }
I20260924 19:05:42.735178 23403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f6f82558166458083b8913ec15e6133 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:05:42.735359 23383 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8493804eaad4c1fba9431b9e2c317d0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c8493804eaad4c1fba9431b9e2c317d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8493804eaad4c1fba9431b9e2c317d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "1f6f82558166458083b8913ec15e6133" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36443 } } peers { permanent_uuid: "4a3bec28013f43878f2e7736da916870" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } } }
I20260924 19:05:42.735572 23383 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8493804eaad4c1fba9431b9e2c317d0 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:42.742879 23403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f6f82558166458083b8913ec15e6133 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8493804eaad4c1fba9431b9e2c317d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8493804eaad4c1fba9431b9e2c317d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "1f6f82558166458083b8913ec15e6133" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36443 } } peers { permanent_uuid: "4a3bec28013f43878f2e7736da916870" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } } }
I20260924 19:05:42.743156 23403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f6f82558166458083b8913ec15e6133 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:05:42.750850 23364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a3bec28013f43878f2e7736da916870 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c8493804eaad4c1fba9431b9e2c317d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8493804eaad4c1fba9431b9e2c317d0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38773 } } peers { permanent_uuid: "1f6f82558166458083b8913ec15e6133" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36443 } } peers { permanent_uuid: "4a3bec28013f43878f2e7736da916870" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45101 } } }
I20260924 19:05:42.751075 23364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a3bec28013f43878f2e7736da916870 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:05:42.797941 23422 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 19:05:42.810395 23398 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:05:42.812250 23398 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:05:42.837822 23398 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8493804eaad4c1fba9431b9e2c317d0: Generated new TSK 0
I20260924 19:05:42.838883 23398 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 19:05:42.926788 23213 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:05:42.910072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48788 (local address 127.0.97.62:38773)
0924 19:05:42.910311 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:42.910318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:42.924497 (+ 14179us) server_negotiation.cc:407] Connection header received
0924 19:05:42.925023 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:42.925029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:42.925137 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:42.925229 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:42.926503 (+ 1274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48788: BlockingRecv error: recv got EOF from 127.0.0.1:48788 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260924 19:05:42.929072 23287 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:05:42.909711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35996 (local address 127.0.97.61:36443)
0924 19:05:42.909914 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:42.909921 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:42.927466 (+ 17545us) server_negotiation.cc:407] Connection header received
0924 19:05:42.927711 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:42.927718 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:42.927815 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:42.927885 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:42.928830 (+ 945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35996: BlockingRecv error: recv got EOF from 127.0.0.1:35996 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 19:05:42.931479 23390 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:05:42.916345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35886 (local address 127.0.97.60:45101)
0924 19:05:42.916554 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:42.916563 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:42.929947 (+ 13384us) server_negotiation.cc:407] Connection header received
0924 19:05:42.930126 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:42.930132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:42.930236 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:42.930351 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:42.931241 (+ 890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35886: BlockingRecv error: recv got EOF from 127.0.0.1:35886 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 19:05:42.938279 23287 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:05:42.936130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36002 (local address 127.0.97.61:36443)
0924 19:05:42.936309 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:42.936315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:42.936768 (+ 453us) server_negotiation.cc:407] Connection header received
0924 19:05:42.936996 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:42.937001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:42.937141 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:42.937248 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:42.938027 (+ 779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36002: BlockingRecv error: recv got EOF from 127.0.0.1:36002 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 19:05:42.944619 23287 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:05:42.942593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36006 (local address 127.0.97.61:36443)
0924 19:05:42.942759 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:42.942765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:42.943220 (+ 455us) server_negotiation.cc:407] Connection header received
0924 19:05:42.943435 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:42.943440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:42.943524 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:42.943631 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:42.944427 (+ 796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36006: BlockingRecv error: recv got EOF from 127.0.0.1:36006 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 19:05:42.957463 23287 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:05:42.955376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36020 (local address 127.0.97.61:36443)
0924 19:05:42.955550 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:42.955557 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:42.956063 (+ 506us) server_negotiation.cc:407] Connection header received
0924 19:05:42.956261 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:42.956267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:42.956367 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:42.956474 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:42.957233 (+ 759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36020: BlockingRecv error: recv got EOF from 127.0.0.1:36020 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260924 19:05:42.966754 23287 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:05:42.964650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36032 (local address 127.0.97.61:36443)
0924 19:05:42.964855 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:42.964862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:42.965337 (+ 475us) server_negotiation.cc:407] Connection header received
0924 19:05:42.965551 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:42.965564 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:42.965650 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:42.965779 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:42.966538 (+ 759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36032: BlockingRecv error: recv got EOF from 127.0.0.1:36032 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260924 19:05:42.980439 23287 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:05:42.978221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36044 (local address 127.0.97.61:36443)
0924 19:05:42.978409 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:42.978415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:42.978930 (+ 515us) server_negotiation.cc:407] Connection header received
0924 19:05:42.979152 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:42.979158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:42.979246 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:42.979343 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:42.980192 (+ 849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36044: BlockingRecv error: recv got EOF from 127.0.0.1:36044 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 19:05:43.005191 23287 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:05:43.003034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36058 (local address 127.0.97.61:36443)
0924 19:05:43.003230 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:43.003236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:43.003750 (+ 514us) server_negotiation.cc:407] Connection header received
0924 19:05:43.003940 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:43.003946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:43.004039 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:43.004147 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:43.004961 (+ 814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36058: BlockingRecv error: recv got EOF from 127.0.0.1:36058 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 19:05:43.044771 23287 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:05:43.042851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36060 (local address 127.0.97.61:36443)
0924 19:05:43.043028 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:43.043033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:43.043471 (+ 438us) server_negotiation.cc:407] Connection header received
0924 19:05:43.043626 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:43.043633 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:43.043739 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:43.043862 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:43.044516 (+ 654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36060: BlockingRecv error: recv got EOF from 127.0.0.1:36060 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 19:05:43.114048 23287 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:05:43.112121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36072 (local address 127.0.97.61:36443)
0924 19:05:43.112330 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:43.112338 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:43.112666 (+ 328us) server_negotiation.cc:407] Connection header received
0924 19:05:43.112845 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:43.112851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:43.112952 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:43.113083 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:43.113810 (+ 727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36072: BlockingRecv error: recv got EOF from 127.0.0.1:36072 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260924 19:05:43.249161 23287 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:05:43.247149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36074 (local address 127.0.97.61:36443)
0924 19:05:43.247403 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:43.247410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:43.247661 (+ 251us) server_negotiation.cc:407] Connection header received
0924 19:05:43.247916 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:43.247922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:43.247999 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:43.248109 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:43.248889 (+ 780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36074: BlockingRecv error: recv got EOF from 127.0.0.1:36074 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 19:05:43.429749 23375 catalog_manager.cc:1259] Loaded cluster ID: 70cedae9573943ac8b0592112f9f6cd7
I20260924 19:05:43.429850 23375 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4a3bec28013f43878f2e7736da916870: loading cluster ID for follower catalog manager: success
I20260924 19:05:43.432392 23375 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4a3bec28013f43878f2e7736da916870: acquiring CA information for follower catalog manager: success
I20260924 19:05:43.434139 23375 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4a3bec28013f43878f2e7736da916870: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 19:05:43.510813 23287 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:05:43.508933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36082 (local address 127.0.97.61:36443)
0924 19:05:43.509212 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:43.509218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:43.509447 (+ 229us) server_negotiation.cc:407] Connection header received
0924 19:05:43.509704 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:43.509714 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:43.509797 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:43.509909 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:43.510636 (+ 727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36082: BlockingRecv error: recv got EOF from 127.0.0.1:36082 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
I20260924 19:05:43.802189 23422 catalog_manager.cc:1259] Loaded cluster ID: 70cedae9573943ac8b0592112f9f6cd7
I20260924 19:05:43.802346 23422 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1f6f82558166458083b8913ec15e6133: loading cluster ID for follower catalog manager: success
I20260924 19:05:43.805119 23422 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1f6f82558166458083b8913ec15e6133: acquiring CA information for follower catalog manager: success
I20260924 19:05:43.807075 23422 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1f6f82558166458083b8913ec15e6133: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:05:44.045215 23489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:44.038022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36090 (local address 127.0.97.61:36443)
0924 19:05:44.038822 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:44.038829 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:44.038851 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:05:44.038935 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:44.038942 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:44.039139 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:44.039301 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:05:44.039642 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:44.040892 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:44.042326 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:44.043086 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:44.043233 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:44.043725 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:05:44.044543 (+ 818us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:05:44.044705 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:44.044778 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 19:05:44.044859 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":495,"threads_started":1}
I20260924 19:05:44.058425 23495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:44.052213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35902 (local address 127.0.97.60:45101)
0924 19:05:44.052920 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:44.052927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:44.052949 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:05:44.053059 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:44.053062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:44.053228 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:44.053402 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:05:44.053825 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:44.054605 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:44.055989 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:44.056464 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:44.056605 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:44.057148 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:05:44.057748 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:05:44.057923 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:44.057982 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 19:05:44.058098 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":647,"threads_started":1}
I20260924 19:05:44.079376 23489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:44.064624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36096 (local address 127.0.97.61:36443)
0924 19:05:44.064806 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:44.064812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:44.071871 (+ 7059us) server_negotiation.cc:407] Connection header received
0924 19:05:44.072351 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:44.072357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:44.072458 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:44.072528 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:05:44.073282 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:44.074430 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:44.076850 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:44.077554 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:44.077658 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:44.078522 (+ 864us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:05:44.078658 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:05:44.078730 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:44.078977 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 19:05:44.079056 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 19:05:44.082456 23502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:44.064465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48798 (local address 127.0.97.62:38773)
0924 19:05:44.074609 (+ 10144us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:44.074615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:44.075038 (+ 423us) server_negotiation.cc:407] Connection header received
0924 19:05:44.075280 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:44.075286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:44.075564 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:44.075739 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:05:44.076384 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:44.077429 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:44.079629 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:44.080423 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:44.080541 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:44.081185 (+ 644us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:05:44.081728 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:05:44.081826 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:44.082058 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 19:05:44.082151 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9722,"thread_start_us":9622,"threads_started":1}
I20260924 19:05:44.082587 23495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:44.064757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35916 (local address 127.0.97.60:45101)
0924 19:05:44.064930 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:44.064939 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:44.075717 (+ 10778us) server_negotiation.cc:407] Connection header received
0924 19:05:44.076600 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:44.076605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:44.076691 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:44.076748 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:05:44.077366 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:44.078437 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:44.080620 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:44.081190 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:44.081301 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:44.081836 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:05:44.082006 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:05:44.082081 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:44.082255 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 19:05:44.082338 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 19:05:44.088193 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 23142
I20260924 19:05:44.089457 23204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:05:44.200702 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 23142
I20260924 19:05:44.222041 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 23216
I20260924 19:05:44.223205 23278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:05:44.327922 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 23216
I20260924 19:05:44.348061 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskLphABj/build/asan/bin/kudu pid 23290
I20260924 19:05:44.349211 23352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:05:44.456007 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLphABj/build/asan/bin/kudu with pid 23290
2026-09-24T19:05:44Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4888 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9645 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (357854 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST